Occupant state determination device, occupant state determination system, occupant state determination method and program

Abstract

A passenger state determination device (1) according to the present disclosure is equipped with: a face detection unit (12); a feature detection unit (13) for detecting a detected passenger posture; a reference posture estimation unit (14) which classifies the detected passenger posture by using a first threshold, and estimates a mode value as the reference posture, which is the detected passenger posture which is most frequently detected by the feature detection unit from among the postures which are obtained during a pre-set interval and which comprise a detected passenger posture which can be viewed as the reference posture of the passenger and a detected passenger posture which can be viewed as a change in the posture of the passenger from the reference posture; a reference posture update determination unit (17) for determining whether or not to update the reference posture by comparing the difference between a first reference posture and a second reference posture with a second threshold; a posture collapse determination unit (18) for determining the type of posture collapse; and an abnormal posture determination unit (19) for determining whether or not the posture of the passenger is abnormal.

Description

[Technical field]

[0001] The present disclosure relates to an occupant state determination device, an occupant state determination system, an occupant state determination method, and a program. [Background technology]

[0002] Technology has been developed that captures images of the inside of a vehicle and uses the facial direction, head position, etc. of the occupants extracted from the captured images to determine whether the occupants have lost consciousness or other abnormalities.It also updates the reference posture that serves as the basis for the occupants' facial direction, head position, etc. if the occupants change their posture intentionally or due to habit (hereinafter referred to as "change in occupant posture") during long periods of driving or due to changes in the driving environment (changes in weather or time of day).

[0003] For example, the information processing device of Patent Document 1 calculates the deviation between the driver's face direction and a reference for the face direction, and if the deviation is within a first range and outside a second range, stores information indicating the deviation in a face direction storage unit by linking it to detection information of the face direction of the image, and if an image showing a deviation from the reference in a certain direction (approximately the same direction) is detected at a predetermined frequency (or ratio), the control unit performs processing to change the reference for the face direction so that the state of the face direction deviated in the certain direction is easily determined to be the front of the driver's face direction, and updates the reference for the face direction of the occupant when the posture of the occupant changes, in other words, when an image showing a deviation from the reference in a certain direction (approximately the same direction) is detected at a predetermined frequency (or ratio). Here, the first range is set to various ranges depending on the purpose of monitoring. The first range is, for example, a range determined by an inattentive judgment angle, a concentration / fatigue degree judgment angle, etc. The second range is an angle range narrower than the first range, and is a range determined by a direction in which the face direction is considered to be the front. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP2019-088522A Summary of the Invention [Problem to be solved by the invention]

[0005] In reality, even if the occupant's posture is in a reference posture and the occupant does not change his / her posture from the reference posture, due to vehicle vibrations, etc., at least one of the occupant's facial direction and head position may move back and forth between at least one of the angles and positions when the occupant is in the reference posture and at least one of the angles and positions when the occupant changes his / her posture (hereinafter referred to as Situation A).

[0006] Therefore, since the posture of the occupant is in the reference posture, the detection frequency A may be higher than the detection frequency B. In addition, during a predetermined period, both the detection frequency (hereinafter referred to as detection frequency A) of at least one of the occupant's facial direction (hereinafter referred to as facial direction A) and head position that can be considered to be in a reference posture and the detection frequency (hereinafter referred to as detection frequency B) of at least one of the occupant's facial direction (hereinafter referred to as facial direction B) and head position that can be considered to have changed the occupant's posture from the reference posture may be equal to or higher than a predetermined frequency. In other words, even though the detection frequency A is higher than the detection frequency B, the detection frequency B may be equal to or higher than the predetermined frequency.

[0007] Here, in the case of situation A, in Patent Document 1, both the detection frequency of the face orientation in an image corresponding to a deviation within the second range corresponding to face orientation A (hereinafter referred to as detection frequency A') and the detection frequency of the face orientation in an image corresponding to a deviation outside the second range corresponding to face orientation B (hereinafter referred to as detection frequency B') may be equal to or higher than a predetermined frequency.

[0008] However, in Patent Document 1, first, it is determined whether the deviation satisfies the condition of being within a first range and outside a second range, and when the face orientation of an image associated with the deviation that satisfies the condition is detected at a predetermined frequency, the standard for the face orientation of the occupant is updated. Therefore, only the detection frequency of the face orientation of an image corresponding to the deviation that satisfies the condition of being within the first range and outside the second range is subject to the determination of whether the standard for the face orientation needs to be changed.

[0009] Therefore, in Patent Document 1, in the case of situation A, even though detection frequency A' is higher than detection frequency B', if detection frequency B' is equal to or higher than a predetermined frequency, detection frequency A' is excluded from the target and only detection frequency B' becomes the target, so even though the actual face direction standard is the face direction having detection frequency A', a process is performed to change the face direction standard to the face direction having detection frequency B'.

[0010] In other words, when trying to update the reference attitude using Patent Document 1, even if the occupant is in the reference attitude, it may be determined that the occupant has changed attitude from the reference attitude, and the occupant's reference attitude may be updated. Therefore, it is difficult to accurately calculate the amount of change from the reference posture used to determine whether or not the posture is out of alignment. As a result, even if the posture is out of alignment, the amount of change cannot be accurately calculated, and therefore the posture is not detected and is determined not to be out of alignment, which causes a problem of lowering the accuracy of detection of the occupant state.

[0011] The present disclosure has been made to solve such problems, and has an object to provide an occupant state determination device that improves the accuracy of detecting the occupant state. [Means for solving the problem]

[0012] The occupant state determination device according to the present disclosure includes a face detection unit that detects the face of a vehicle occupant in an image captured of the face of the occupant, a feature detection unit that detects a detected occupant posture, which is at least one of the occupant's facial direction and head position, using the occupant's face in the image detected by the face detection unit, and when the detected occupant posture detected by the feature detection unit is less than a first threshold, the detected occupant posture can be regarded as a reference posture, which is at least one of the facial direction and head position serving as a reference for the occupant, and when the detected occupant posture is equal to or greater than the first threshold, the detected occupant posture can be regarded as a detected occupant posture that can be regarded as a posture change of the occupant from the reference posture, and an occupant posture acquired during a predetermined period is regarded as being the reference posture. a reference posture estimation unit that estimates, as a reference posture, a most frequent detected occupant posture that is most frequently detected by the feature detection unit among detected occupant postures that can be regarded as having been changed from a reference posture by the occupant; a reference posture update determination unit that updates the first reference posture to the second reference posture when a difference between a first reference posture that is a reference posture estimated using the most frequent value in a captured image and a second reference posture that is a reference posture estimated using the most frequent value in a captured image different from the captured image is equal to or greater than a second threshold, and does not update the first reference posture when the difference is less than the second threshold; and a reference posture update determination unit that determines whether the posture of the occupant falls into any of a plurality of posture collapse types by comparing the detected occupant posture with the first reference posture. mosquito a posture collapse determination unit that determines whether the posture of the occupant corresponds to one of a plurality of posture collapse types; mosquito If it is determined that the occupant is in an abnormal posture, the posture deviation determination unit determines that the occupant's posture falls into any one of a plurality of posture deviation types. too and an abnormal attitude determination unit that determines that the occupant is not in an abnormal attitude if it is determined that this is not the case.

[0013] The occupant state determination system according to the present disclosure includes a face detection unit that detects the face of a vehicle occupant in an image captured of the face of the occupant, a feature detection unit that detects a detected occupant posture, which is at least one of the occupant's facial direction and head position, using the occupant's face in the image detected by the face detection unit, and when the detected occupant posture detected by the feature detection unit is less than a first threshold, the detected occupant posture can be considered to be a reference posture, which is at least one of the facial direction and head position that serve as a reference for the occupant, and when the detected occupant posture is equal to or greater than the first threshold, the detected occupant posture can be considered to be a detected occupant posture that can be considered to have changed from the reference posture, and a posture acquired during a predetermined period that the occupant considers to be the reference posture. a reference posture estimation unit that estimates as a reference posture a most frequent detected occupant posture that is most frequently detected by the feature detection unit among detected occupant postures that can be regarded as having been changed from a reference posture by the occupant; a reference posture update determination unit that updates the first reference posture to the second reference posture when a difference between a first reference posture that is a reference posture estimated using the most frequent value in a captured image and a second reference posture that is a reference posture estimated using the most frequent value in a captured image different from the captured image is equal to or greater than a second threshold value, and does not update the first reference posture when the difference is less than the second threshold value; and a reference posture update determination unit that determines whether the posture of the occupant falls into any of a plurality of posture collapse types by comparing the detected occupant posture with the first reference posture. mosquito a posture collapse determination unit that determines whether the posture of the occupant corresponds to one of a plurality of posture collapse types; mosquito If it is determined that the occupant is in an abnormal posture, the posture deviation determination unit determines that the occupant's posture falls into any one of a plurality of posture deviation types. too The occupant state determination device includes an abnormal posture determination unit that determines that the occupant is not in an abnormal posture when it is determined that the occupant is not in an abnormal posture, an imaging device that is installed in the vehicle so as to be able to image at least the face of the occupant, and an output device that is mounted in the vehicle and outputs a warning that the posture of the occupant is an abnormal posture. Occupant status In the judgment system, the occupant status judgment device judges whether the occupant's posture is abnormal or not using an image captured by an imaging device, and if it is determined that the occupant's posture is abnormal, outputs a warning that the occupant's posture is abnormal from an output device.

[0014] The occupant state determination method according to the present disclosure includes a step in which a face detection unit detects the face of a vehicle occupant in an image in which the face of the occupant is captured, a step in which a feature detection unit detects a detected occupant posture, which is at least one of the occupant's facial direction and head position, using the occupant's face in the image detected by the face detection unit, and a step in which a reference posture estimation unit determines, when the detected occupant posture detected by the feature detection unit is less than a first threshold, that the detected occupant posture can be regarded as a reference posture, which is at least one of the occupant's facial direction and head position serving as a reference for the occupant, and, when the detected occupant posture is equal to or greater than the first threshold, that the detected occupant posture can be regarded as a detected occupant posture that can be regarded as a posture change of the occupant from the reference posture, and estimates, when the detected occupant posture detected by the feature detection unit is less than a first threshold, that the detected occupant posture can be regarded as a posture change of the occupant from the reference posture, a step of estimating, as a reference posture, a most frequent detected occupant posture that is most frequently detected by the feature detection unit among detected occupant postures that can be regarded as being a posture that can be considered to be a posture that the occupant has changed from the reference posture; a step of a reference posture update determination unit updating the first reference posture to the second reference posture when a difference between a first reference posture that is a reference posture estimated using the most frequent value in the captured image and a second reference posture that is a reference posture estimated using the most frequent value in a captured image different from the captured image is equal to or greater than a second threshold, and not updating the first reference posture when the difference is less than the second threshold; and a posture collapse determination unit determining whether the posture of the occupant falls into any of a plurality of posture collapse types by comparing the detected occupant posture with the first reference posture. mosquito and a step of determining whether the posture of the occupant corresponds to any of a plurality of posture deviation types, mosquito If it is determined that the occupant is in an abnormal posture, the posture deviation determination unit determines that the occupant's posture falls into any one of a plurality of posture deviation types. too and if it is determined that this is not the case, determining that the occupant is not in an abnormal posture.

[0015] The program according to the present disclosure includes a step in which a face detection unit detects the face of a vehicle occupant in an image in which the face of the occupant is captured, a step in which a feature detection unit detects a detected occupant posture, which is at least one of the occupant's facial direction and head position, using the occupant's face in the image detected by the face detection unit, and a step in which a reference posture estimation unit determines, when the detected occupant posture detected by the feature detection unit is less than a first threshold, that the detected occupant posture can be regarded as a reference posture, which is at least one of the occupant's facial direction and head position serving as a reference for the occupant, and, when the detected occupant posture is equal to or greater than the first threshold, that the detected occupant posture can be regarded as a detected occupant posture that can be regarded as a posture change of the occupant from the reference posture, and determines, when the detected occupant posture detected by the feature detection unit is less than a first threshold, that the detected occupant posture can be regarded as a posture change of the occupant from the reference posture, and determines, when the detected occupant posture is equal to or greater than the first threshold, that the occupant posture can be regarded as a posture change of the occupant from the reference posture, the detected occupant posture obtained during a predetermined period. a step of estimating, as a reference posture, a most frequent detected occupant posture that is most frequently detected by the feature detection unit among detected occupant postures that can be assumed to be changed by the occupant from the reference posture and detected occupant postures that can be assumed to be changed by the occupant from the reference posture; a step of a reference posture update determination unit updating the first reference posture to the second reference posture when a difference between a first reference posture that is a reference posture estimated using the most frequent value in the captured image and a second reference posture that is a reference posture estimated using the most frequent value in a captured image different from the captured image is equal to or greater than a second threshold, and not updating the first reference posture when the difference is less than the second threshold; and a step of a posture collapse determination unit determining whether the posture of the occupant falls into any of a plurality of posture collapse types by comparing the detected occupant posture with the first reference posture. mosquito and a step of determining whether the posture of the occupant corresponds to any of a plurality of posture deviation types, mosquito If it is determined that the occupant is in an abnormal posture, the posture deviation determination unit determines that the occupant's posture falls into any one of a plurality of posture deviation types. too If it is determined that this is not the case, the computer executes a step of determining that the occupant is not in an abnormal posture. Effect of the Invention

[0016] According to the occupant state determination device according to the embodiment of the present disclosure, it is possible to improve the accuracy of detecting the occupant state. [Brief description of the drawings]

[0017] [Figure 1] 1 is a diagram illustrating an example of the configuration of an occupant state determination device according to a first embodiment. [Diagram 2] 5 is a diagram showing a relationship between a detected driver attitude and a detection frequency of each value of the detected driver attitude in a predetermined period according to the first embodiment. FIG. [Diagram 3] 5 is a diagram showing a relationship between a detected driver attitude and a detection frequency of each value of the detected driver attitude in a predetermined period according to the first embodiment. FIG. [Figure 4] 5 is a diagram showing a relationship between a detected driver attitude and a detection frequency of each value of the detected driver attitude in a predetermined period according to the first embodiment. FIG. [Diagram 5] 11A and 11B are diagrams for explaining a plurality of types of poor posture. [Figure 6] 5 is a flowchart showing an example of the operation of the occupant state determination device according to the first embodiment. [Figure 7] 1 is a diagram illustrating an example of a hardware configuration of an occupant state determination device according to a first embodiment. [Figure 8] 1 is a diagram illustrating an example of a hardware configuration of an occupant state determination device according to a first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1 The occupant state determination device according to the first embodiment determines whether or not the posture of a vehicle occupant is abnormal, based on a captured image in which at least the face of the vehicle occupant is captured. In the following first embodiment, as an example, the vehicle occupant for which the occupant state determination device determines whether or not the vehicle is in an abnormal posture is the driver of the vehicle.

[0019] FIG. 1 is a diagram showing an example of the configuration of an occupant state determination device 1 according to the first embodiment. The occupant state determination device 1 according to the first embodiment is assumed to be mounted in a vehicle. The occupant state determination device 1 is connected to an imaging device 2 and an output device 3, and the occupant state determination device 1, the imaging device 2, and the output device 3 constitute an occupant state determination system 100. When the driver changes his / her posture intentionally or due to habit due to long driving time or change in driving environment (change in weather or time of day), the occupant state determination device 1 updates the reference posture, which is the driver's reference facial direction and head position.

[0020] The imaging device 2 is mounted on a vehicle and installed so as to be able to capture at least an area in which the driver's face should be present. In the first embodiment, it is assumed that the imaging device 2 is installed, for example, near the center of the instrument panel in the vehicle width direction or in the center console. For example, the imaging device 2 may be shared with a so-called DMS (Driver Monitoring System) that is installed for the purpose of monitoring the interior of the vehicle. The imaging device 2 is a visible light camera or an infrared camera. The imaging device 2 outputs the captured image to the occupant state determination device 1. The occupant state determination device 1 determines whether or not the driver's posture is abnormal, using the captured image captured by the imaging device 2. The occupant state determination device 1 will be described in detail later. When the occupant state determination device 1 determines that the driver's posture is abnormal, it causes the output device 3 to output a warning that the driver's posture is abnormal.

[0021] The output device 3 is mounted on the vehicle and outputs a warning that the driver's posture is abnormal. The output device 3 is, for example, an audio output device such as a speaker. The output device 3 may be provided in, for example, an audio device provided in a vehicle. For example, when information for outputting a warning (hereinafter referred to as "warning information") is output from the occupant state determination device 1, the output device 3 outputs a warning sound or a voice message notifying that the driver's posture is abnormal. The output device 3 may also be, for example, a display device such as a display. For example, when warning information is output from the occupant state determination device 1, the output device 3 displays a message notifying that the driver's posture is abnormal. The output device 3 may be, for example, a turn signal, a hazard lamp, or a headlight provided on the exterior of a vehicle so as to be visible from other vehicles. For example, the occupant state determination device 1 may output warning information to the output device 3 to notify people outside the vehicle, such as occupants of other vehicles, that the driver of the vehicle is in an abnormal state. The output device 3 may be mounted on, for example, another vehicle. The occupant state determination device 1 can transmit warning information to the output device 3 mounted on the other vehicle and output a voice or display to inform the occupants of the other vehicle that the driver of the vehicle is in an abnormal state.

[0022] As shown in FIG. 1, the occupant state determination device 1 includes an image acquisition unit 11, a face detection unit 12, a feature detection unit 13, a reference posture estimation unit 14, a reference posture memory unit 15, a reference posture candidate memory unit 16, a reference posture update determination unit 17, a posture collapse determination unit 18, an abnormal posture determination unit 19, and an output control unit 20. The feature detection unit 13 includes a face direction detection unit 131 , a head position detection unit 132 , and a detection reliability calculation unit 133 .

[0023] The image acquisition unit 11 acquires a captured image from the imaging device 2 . The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12.

[0024] The face detection unit 12 detects the face of the driver in the captured image acquired by the image acquisition unit 11 . In detail, the face detection unit 12 detects the driver's facial feature points indicating the driver's facial parts by using a known image recognition technique such as edge detection for the captured image. The facial parts are the corners and corners of the eyes, the nose, the mouth, the eyebrows, the chin, etc. The driver's facial feature points are represented, for example, by coordinates on the captured image. The face detection unit 12 may also detect the driver's facial area. The driver's facial area is, for example, the smallest rectangle that encloses the outline of the driver's face. The driver's facial area is, for example, represented by the coordinates of the four corners of the smallest rectangle on the captured image. Since the installation position and the angle of view of the imaging device 2 are known in advance, even if a plurality of passengers are captured in the captured image, the face detection unit 12 can determine which area of ​​the captured image is the driver's face. For example, an area in the captured image where the driver's face may exist (hereinafter referred to as the "driver detection area") is set in advance, and the face detection unit 12 detects the driver's facial feature points and face area using a known image recognition technique in the driver detection area. The face detection unit 12 outputs information about the detected face of the driver (hereinafter referred to as "face information") to the feature detection unit 13. The face information is, for example, a captured image to which information capable of identifying the feature points and face area of ​​the driver's face is added.

[0025] The feature detection unit 13 detects features for determining the posture of an occupant, in this case, the driver. The feature for determining the driver's posture is at least one of the driver's facial direction and the driver's head position (hereinafter referred to as the detected driver posture).

[0026] The face direction detection unit 131 detects the face direction of the driver in real space based on the face information output from the face detection unit 12, specifically, based on the face of the driver in the captured image detected by the face detection unit 12. The face direction detection unit 131 may detect the face direction of the driver using, for example, a known face direction detection technique for detecting the face direction from the captured image. The face direction of the driver is represented, for example, by at least one of a yaw angle, which is a rotation angle about an axis in the vertical direction of the vehicle, a pitch angle, which is a rotation angle about an axis in the horizontal direction of the vehicle, and a roll angle, which is a rotation angle about an axis in the longitudinal direction of the vehicle, which are predetermined references. The face direction detection unit 131 outputs information on the detected face direction of the driver (hereinafter referred to as "face direction information") to the reference posture estimation unit 14 and the posture collapse determination unit 18. The face direction information includes at least one of information on the yaw angle, the pitch angle, and the roll angle.

[0027] The head position detection unit 132 detects the head position of the driver in real space based on the face information output from the face detection unit 12, more specifically, based on the face of the driver in the captured image detected by the face detection unit 12. In the first embodiment, the head position of the driver in the captured image is indicated, for example, by the center of the driver's eyebrows. The head position detection unit 132 detects, for example, a point in real space corresponding to the center of the driver's eyebrows in the captured image as the driver's head position. Note that this is merely an example, and the position of the driver's head in the captured image may be indicated, for example, by the center of the driver's face area or the center of a line connecting the driver's eye corners. In this case, the head position detection unit 132 detects, for example, a point in real space corresponding to the center of the driver's face area in the captured image or the center of a line connecting the driver's eye corners as the driver's head position in real space.

[0028] Head position detection unit 132 may detect the driver's head position, for example, using a known coordinate conversion technique that converts points on a captured image into points in real space. The driver's head position is represented, for example, by coordinates in real space. Head position detection unit 132 outputs information related to the detected driver's head position (hereinafter referred to as "head position information") to reference posture estimation unit 14 and posture collapse determination unit 18. The head position information includes coordinate information of the driver's head position.

[0029] The detection reliability calculation unit 133 calculates a detection reliability indicating a reliability that the detected driver posture is at least one of a predetermined face direction and a head position. Then, the detection reliability calculation unit 133 outputs the calculated detection reliability to at least one of the reference posture estimation unit 14 and the posture collapse determination unit 18. For example, a method shown in the document "Rapid Object Detection using a Boosted Cascade of Simple Features", 2001, Paul Viola, et al. can be used as a method for calculating the detection reliability. In this document, the detection reliability of at least one of the face direction and the head position is calculated from the degree of match in matching between at least one of the face direction and the head position learned in advance from a large number of face images and at least one of the face direction and the head position extracted from an image of the driver.

[0030] In addition, even if the method is not the method described in the above document, as long as it is a method that can detect a specific object corresponding to at least one of the face direction and head position from an image and calculate the detection reliability, the detection reliability calculation unit 133 can use that method.

[0031] When the detected driver posture is less than a first threshold, the reference posture estimation unit 14 determines that the detected driver posture can be regarded as a reference posture (hereinafter referred to as detected driver posture A), and when the detected driver posture is equal to or greater than the first threshold, the reference posture estimation unit 14 determines that the detected driver posture can be regarded as a posture that the driver has changed from the reference posture (hereinafter referred to as detected driver posture B).

[0032] Then, the most frequent detected driver posture among detected driver posture A and detected driver posture B during a predetermined period (hereinafter referred to as the "reference setting period") is estimated as the driver's reference posture by the feature detection unit 13.

[0033] The reference set period is, for example, a period from when the vehicle ignition is turned on until a preset time (for example, 10 seconds) has elapsed. Note that this is merely an example, and the reference set period may be, for example, a predetermined time (for example, 3 seconds) when the vehicle speed is equal to or greater than a preset speed (for example, 25 [km / h]) and the steering angle is within a preset range (for example, ±20 degrees).

[0034] In addition, the detected driver posture is used to estimate at least one of the driver's reference facial direction and head position as the reference posture, but the reference posture may also be estimated using the detected driver posture and at least one of the detection reliability of the facial direction and the detection reliability of the head position.

[0035] The reference posture estimation unit 14 outputs a reference posture (hereinafter, the first reference posture) estimated using the most frequent value in the captured image to the reference posture memory unit 15, and after outputting the first reference posture to the reference posture memory unit 15, the reference posture estimation unit 14 outputs a reference posture (hereinafter, the second reference posture) estimated using the most frequent value in an captured image different from the captured image from which the first reference posture output to the reference posture memory unit 15 was estimated to the reference posture candidate memory unit 16. Furthermore, the reference attitude estimation unit 14 acquires the first reference attitude from the reference attitude storage unit 15 as the reference attitude to be output to the attitude collapse determination unit 18 .

[0036] The face direction in the first reference posture and the second reference posture is expressed by at least one of a yaw angle, a pitch angle, and a roll angle. The head position in the first reference posture and the second reference posture is expressed by, for example, coordinates in a real space. The reference posture information includes coordinate information of the driver's head position as information of the driver's head position.

[0037] FIG. 2 shows a graph showing the relationship between the detected driver posture and the frequency with which each value of the detected driver posture is detected by the feature detection unit 13 during a predetermined period (hereinafter referred to as the detection frequency), which is used when the reference posture estimation unit 14 estimates the driver's reference posture.

[0038] The vertical axis indicates the detection frequency, which takes values ​​from 0 to 35 in Fig. 2. The horizontal axis indicates the detected driver posture, which takes values ​​from 150 to 250 in Fig. 2. Here, the detected driver posture in Fig. 2 refers to the head position of the driver detected by feature detection unit 13. Furthermore, when the driver is in a specific head position, the detection frequency of the specific head position will be higher than other head positions. Here, the detected driver posture is classified into detected driver posture A and detected driver posture B. For example, if the first threshold is 240, when the detected driver posture is less than the first threshold, it is classified as detected driver posture A, and when the detected driver posture is equal to or greater than the first threshold, it is classified as detected driver posture B. In FIG. 2, the detected driver posture A is 150-230, and the detected driver posture B is 240-250.

[0039] The reference posture estimation unit 14 estimates the most frequent value of the detected driver posture as a first reference posture from among the detected driver postures A and B shown in Fig. 2. In Fig. 2, the reference posture estimation unit 14 estimates 190 as the first reference posture and outputs the estimated first reference posture to the reference posture storage unit 15.

[0040] The reference attitude storage unit 15 stores the first reference attitude output from the reference attitude estimation unit 14. When the reference attitude update determination unit 17, which will be described later, determines to update the first reference attitude to the second reference attitude, the first reference attitude is updated to the second reference attitude.

[0041] The reference attitude candidate storage unit 16 stores the second reference attitude output from the reference attitude estimation unit .

[0042] a reference attitude update determination unit that updates the first reference attitude to the second reference attitude when a difference between the first reference attitude stored in the reference attitude storage unit and the second reference attitude stored in the reference attitude candidate storage unit is equal to or greater than a second threshold value; If the difference is less than the second threshold, the first reference attitude is not updated.

[0043] FIG. 3 shows a graph illustrating the relationship between the detected driver posture and the detection frequency of each value of the detected driver posture in a predetermined period when the driver changes his posture from the reference posture after the reference posture estimation unit 14 estimates the reference posture in the graph shown in FIG. 2. The vertical axis indicates the detection frequency, which takes values ​​from 0 to 35 in FIG. 3. The horizontal axis indicates the detected driver posture, which takes values ​​from 150 to 250 in FIG. 3. Here, the detected driver posture in FIG. 3 refers to the head position of the driver detected by the feature detection unit 13. Furthermore, when the driver is in a specific head position, the detection frequency of the specific head position is higher than other head positions. Furthermore, similar to FIG. 2, the detected driver posture is classified as detected driver posture A or detected driver posture B.

[0044] The reference attitude update determination by the reference attitude update determination unit 17 in the situation shown in FIG. 3 will be described. In Fig. 3, the reference posture estimation unit 14 estimates the most frequent value of the detected driver posture as a first reference posture from among the detected driver postures A and B shown in Fig. 3. In Fig. 3, the reference posture estimation unit 14 outputs 240 to the reference posture candidate storage unit 16 as a second reference posture.

[0045] Here, the reference attitude update determination unit 17 determines whether or not to update the first reference attitude by comparing the difference between the first reference attitude stored in the reference attitude memory unit 15 and the second reference attitude stored in the reference attitude candidate memory unit 16 with a second threshold value.

[0046] For example, if the second threshold is 45, when the difference is equal to or greater than the second threshold, the first reference attitude is updated to the second reference attitude. When the difference is less than the second threshold, the first reference attitude is not updated. Here, when the first reference attitude is 190 and the second reference attitude is 240, the difference is 50, which satisfies the condition of being equal to or greater than the second threshold. Therefore, the first reference attitude is updated to the second reference attitude 240.

[0047] FIG. 4 shows a graph illustrating the relationship between the detected driver posture in situation A and the detection frequency of each value of the detected driver posture in a predetermined period after the reference posture estimation unit 14 estimates the reference posture in the graph shown in FIG. The vertical axis indicates the detection frequency, which takes values ​​from 0 to 35 in FIG. 4. The horizontal axis indicates the detected driver posture, which takes values ​​from 150 to 250 in FIG. 4. Here, the detected driver posture in FIG. 4 refers to the head position of the driver detected by the feature detection unit 13. Also, when the driver is in a specific head position, the detection frequency of the specific head position is higher than other head positions. Also, similar to FIG. 2, the detected driver posture is classified as detected driver posture A or detected driver posture B. Here, situation A refers to a situation in which, even if the driver's posture is actually located in a reference posture and the driver does not change his / her posture from the reference posture, the detected driver posture fluctuates between detected driver posture A and detected driver posture B due to vehicle vibration, etc.

[0048] In the case of situation A, the detection frequency of detected driver posture A (detection frequency of 230 in FIG. 4) and the detection frequency of detected driver posture B (detection frequency of 240 in FIG. 4) may both be equal to or higher than a predetermined frequency. In addition, in the case of situation A, the detection frequency of detected driver posture A (detection frequency of 230 in FIG. 4) may be higher than the detection frequency of detected driver posture B (detection frequency of 240 in FIG. 4). In other words, even if the detection frequency of the detected driver posture A is higher than the detection frequency of the detected driver posture B, the detection frequency of the detected driver posture B may be equal to or higher than the predetermined frequency.

[0049] The reference attitude update determination by the reference attitude update determination unit 17 in the situation shown in FIG. 4 will be described. The reference posture estimation unit 14 estimates the most frequent value of the detected driver posture as a second reference posture from among the detected driver postures A and B shown in Fig. 4. In Fig. 4, the reference posture estimation unit 14 estimates 230 as the second reference posture and outputs the estimated second reference posture to the reference posture candidate storage unit 16.

[0050] Here, the reference attitude update determination unit 17 determines whether or not to update the first reference attitude by comparing the difference between the first reference attitude stored in the reference attitude memory unit 15 and the second reference attitude stored in the reference attitude candidate memory unit 16 with a second threshold value.

[0051] For example, if the second threshold is 45, when the difference is equal to or greater than the second threshold, the first reference attitude is updated to the second reference attitude. When the difference is less than the second threshold, the first reference attitude is not updated. Here, when the first reference attitude is 190 and the second reference attitude is 230, the difference is 40, which does not satisfy the condition of being equal to or greater than the second threshold. Therefore, the first reference attitude of 190 is not updated.

[0052] The posture imbalance determination unit 18 determines whether or not the driver's posture corresponds to any one of a plurality of predetermined posture imbalance types (hereinafter referred to as "posture imbalance types") by comparing the detected driver posture with the reference posture acquired from the reference posture estimation unit 14. In the first embodiment, the determination by the posture imbalance determination unit 18 of whether the driver's posture corresponds to any one of a plurality of posture imbalance types is also referred to as "posture imbalance type determination." The plurality of posture deviation types include types of postures that are determined to be abnormal postures caused by the driver being in an abnormal state.

[0053] Here, FIG. 5 (partially modified from the "Basic Design Document for Driver Abnormality Automatic Detection System" by the Advanced Safety Vehicle Promotion Study Group, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, March 2018) is a diagram for explaining multiple types of posture imbalance. As shown in FIG. 5, there are multiple types of posture imbalance. FIG. 5 shows "head down" in which the driver falls forward and continues to face near the steering wheel, "head down" in which the driver's face continues to face down, "backward lean" in which the driver's upper body is tilted backward and the face continues to face up, "backward lean" in which the driver's upper body is arched up and the face continues to face up, "head down" in which the driver's ... In the present disclosure, the multiple pre-defined types of posture problems are "face down", "head down", "backward arch", "arched back", "head only sideways", "sideways", and "leaning" shown in Fig. 5. Information on the multiple types of posture problems is stored in a location that can be referenced by the posture problem determination unit 18.

[0054] In detail, the posture deviation judgment unit 18 judges which of a plurality of posture deviation types the driver belongs to depending on how much the detected driver posture has changed compared to a first reference posture. Note that the condition for judging which posture deviation type the driver belongs to depending on how much the detected driver posture has changed compared to the first reference posture is preset and held by the posture deviation judgment unit 18. The posture deviation judgment unit 18 may perform the posture deviation type judgment using a learning device such as SVM (Support Vector Machine), for example.

[0055] The posture deviation determination unit 18 outputs the posture deviation type determination result to the abnormal posture determination unit 19. The posture deviation type determination result includes information on whether or not the driver's posture corresponds to any of the posture deviation types.

[0056] The abnormal posture determination unit 19 uses the posture error type determination result output by the posture error determination unit 18 to determine whether or not the driver's posture is abnormal.

[0057] When the posture error determination unit 18 determines that the posture of the driver does not correspond to any of the posture error types, the abnormal posture determination unit 19 determines that the driver is not in an abnormal posture.

[0058] When the posture error determination unit 18 determines that the driver's posture is an abnormal posture, the abnormal posture determination unit 19 determines the duration of the state in which the driver's posture is an abnormal posture. For example, the abnormal posture determination unit 19 stores the acquired range determination result or posture error type determination result in a storage unit (not shown) in association with information on the date and time when the range determination result or posture error type determination result was acquired. The abnormal posture determination unit 19 may determine the duration of the state in which the driver's posture is an abnormal posture from the stored range determination result or posture error type determination result. When the duration of the state in which the driver's posture is an abnormal posture reaches a preset threshold (hereinafter referred to as "abnormal posture determination threshold"), the abnormal posture determination unit 19 determines that the driver has an abnormal posture.

[0059] The abnormal posture determination unit 19 outputs to the output control unit 20 a determination result as to whether or not the driver is in an abnormal posture (hereinafter referred to as an “abnormal posture determination result”).

[0060] When the abnormal posture determination unit 19 determines that the driver is in an abnormal posture, the output control unit 20 outputs, to the output device 3, warning information for warning the driver.

[0061] A description will be given of the operation of the occupant state determination device 1 according to the embodiment 1. Fig. 6 is a flowchart for explaining the operation of the occupant state determination device 1 according to the embodiment 1. The operation of the occupant state determination device 1 is started, for example, after the ignition of the vehicle is turned on. It is preferable to perform the process of determining the occupant state every time an image is acquired from the imaging device 2, that is, for each frame.

[0062] The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST1). The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12.

[0063] The face detection unit 12 detects the face of the driver in the captured image acquired by the image acquisition unit 11 in step ST1 (step ST2). The face detection unit 12 outputs the face information to the feature detection unit 13 .

[0064] The face direction detection unit 131 detects the face direction of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The face direction detection unit 131 outputs the face direction information to the reference posture estimation unit 14 and posture deviation determination unit 18.

[0065] The head position detection unit 132 detects the head position of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The head position detection unit 132 outputs the head position information to the reference posture estimation unit 14 and posture deviation determination unit 18.

[0066] The head position detection unit 132 detects the head position of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The head position detection unit 132 outputs the head position information to the reference posture estimation unit 14 and posture deviation determination unit 18.

[0067] The detection reliability calculation unit 133 calculates the detection reliability of at least one of the face direction and the head position based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST4). The detection reliability calculation unit 133 outputs the detection reliability to at least one of the reference posture estimation unit 14 and the posture collapse determination unit 18.

[0068] The reference posture estimation unit 14 determines the detected driver posture as detected driver posture A when the detected driver posture is less than the first threshold, and determines the detected driver posture as detected driver posture B when the detected driver posture is equal to or greater than the first threshold (step ST5).

[0069] The reference posture estimation unit 14 estimates the most frequent detected driver posture, which is one of the detected driver postures A and B during the reference setting period and which is detected most frequently by the feature detection unit 13, as the driver's reference posture (step ST6).

[0070] If the reference posture estimation unit 14 has completed estimating the driver's reference posture in the previous frame, in other words, after outputting the first reference posture to the reference posture memory unit 15 (if "YES" in step ST7), the reference posture estimation unit 14 outputs a second reference posture estimated using the most frequent value in an image different from the image used to estimate the first reference posture output to the reference posture memory unit 15 to the reference posture candidate memory unit 16 (step ST8).

[0071] If the reference posture estimation unit 14 has not completed estimation of the driver's reference posture in the previous frame (if "NO" in step ST7), the reference posture estimation unit 14 outputs the first reference posture in the reference posture estimated using the most frequent value in the captured image to the reference posture memory unit 15 (step ST9).

[0072] When the reference posture estimation unit 14 estimates the reference posture of the driver, it sets a reference posture estimation completion flag to "1". The initial value of the reference posture estimation completion flag is "0", and the reference posture estimation completion flag is initialized, for example, when the ignition of the occupant state determination device 1 is turned off. The reference posture estimation completion flag is stored in a location that can be referenced by the occupant state determination device 1 (step ST10).

[0073] If the difference between the first reference posture stored in the reference posture memory unit 15 and the second reference posture stored in the reference posture candidate memory unit 16 is greater than or equal to a second threshold value (if "YES" in step ST11), the reference posture update determination unit 17 updates the first reference posture to the second reference posture (step ST12).

[0074] When the difference between the first reference attitude and the second reference attitude is less than the second threshold value ("NO" in step ST11), the reference attitude update determination unit 17 does not update the first reference attitude (step ST13).

[0075] The posture deviation determination unit 18 determines whether the driver's posture corresponds to any one of a plurality of predetermined posture deviation types by comparing the detected driver's posture with the reference posture obtained from the reference posture estimation unit 14 (step ST14). The posture imbalance determination unit 18 outputs the posture imbalance type determination result to the abnormal posture determination unit 19.

[0076] If the posture error determination unit 18 determines that the driver's posture does not fall into any of the posture error types ("NO" in step ST14), the abnormal posture determination unit 19 determines that the driver is not in an abnormal posture (step ST15).

[0077] When the posture imbalance determination unit 18 determines that the driver's posture is imbalance ("YES" in step ST14), the abnormal posture determination unit 19 determines the duration of the state in which the driver's posture is imbalance (step ST16).

[0078] If the duration of the driver's posture being out of alignment reaches the abnormal posture determination threshold value (if "YES" in step ST16), the abnormal posture determination unit 19 determines that the driver is in an abnormal posture and causes the output device 3 to output a warning that the driver's posture is abnormal (step ST17). If the duration of the state in which the driver's posture is out of alignment does not reach the abnormal posture determination threshold value ("NO" in step ST16), the abnormal posture determination unit 19 determines that the driver is not in an abnormal posture.

[0079] 7 and 8 are diagrams showing an example of a hardware configuration of the occupant state determination device 1 according to the first embodiment. In the first embodiment, the functions of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference posture estimation unit 14, the reference posture update determination unit 17, the posture collapse determination unit 18, the abnormal posture determination unit 19, and the output control unit 20 are realized by a processing circuit 1001. That is, the occupant state determination device 1 includes a processing circuit 1001 for performing control to determine whether the posture of the vehicle occupant is an abnormal posture based on a captured image. The processing circuit 1001 may be dedicated hardware as shown in FIG. 7, or may be a processor 1004 that executes a program stored in a memory 1005 as shown in FIG. 8. The memory 1000 includes the reference posture estimation unit 14 and the reference posture storage unit 15.

[0080] When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0081] When the processing circuit is the processor 1004, the functions of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference posture estimation unit 14, the reference posture update determination unit 17, the posture collapse determination unit 18, the abnormal posture determination unit 19, and the output control unit 20 are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 1005. The processor 1004 executes the functions of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference posture estimation unit 14, the reference posture update determination unit 17, the posture collapse determination unit 18, the abnormal posture determination unit 19, and the output control unit 20 by reading and executing the program stored in the memory 1005. That is, the occupant state determination device 1 includes a memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST16 in FIG. 7 described above. Also, it can be said that the programs stored in the memory 1005 cause the computer to execute the procedures or methods of the processes of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference posture estimation unit 14, the reference posture update determination unit 17, the posture collapse determination unit 18, the abnormal posture determination unit 19, and the output control unit 20. Here, the memory 1005 corresponds to, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), or the like.

[0082] The functions of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, posture collapse determination unit 18, abnormal posture determination unit 19, and output control unit 20 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the functions of the image acquisition unit 11 and the output control unit 20 may be realized by a processing circuit 1001 as dedicated hardware, and the functions of the face detection unit 12, feature detection unit 13, reference posture estimation unit 14, posture collapse determination unit 18, and abnormal posture determination unit 19 may be realized by a processor 1004 reading out and executing a program stored in a memory 1005.

[0083] The storage unit (not shown) is, for example, a memory 1005. The occupant state determination device 1 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the imaging device 2 or the output device 3.

[0084] As described above, according to the first embodiment, the occupant state determination device 1 includes a face detection unit 12 that detects the face of a vehicle occupant in an image in which the face of the occupant is captured, a feature detection unit 13 that detects a detected occupant posture, which is at least one of the facial direction and head position of the occupant, using the face of the occupant in the image detected by the face detection unit 12, and a mode that is a detected occupant posture that can be considered to be a reference posture, which is at least one of a facial direction and a head position serving as a reference for the occupant, when the detected occupant posture detected by the feature detection unit 13 is less than a first threshold value, and that can be considered to be a detected occupant posture that can be considered to be a posture changed by the occupant from the reference posture, when the detected occupant posture is equal to or greater than the first threshold value, and a mode that is a detected occupant posture that is most frequently detected by the feature detection unit 13 among the detected occupant postures that can be considered to be the reference posture and the detected occupant postures that can be considered to be a posture changed by the occupant from the reference posture, which are acquired during a predetermined period. as a reference posture; a reference posture update determination unit 17 which updates the first reference posture to the second reference posture if a difference between a first reference posture, which is a reference posture estimated using the most frequent value in the captured image, and a second reference posture, which is a reference posture estimated using the most frequent value in an captured image different from the captured image, is equal to or greater than a second threshold, and does not update the first reference posture if the difference is less than the second threshold; a posture collapse determination unit 18 which performs a posture collapse type determination as to which of a plurality of posture collapse types the posture of the occupant falls under by comparing the detected occupant posture with the first reference posture; and an abnormal posture determination unit 19 which determines that the occupant has an abnormal posture if the posture collapse determination unit 18 determines that the posture of the occupant falls under any of the plurality of posture collapse types, and determines that the occupant does not have an abnormal posture if the posture collapse determination unit 18 determines that the posture of the occupant does not fall under any of the plurality of posture collapse types.

[0085] The reference posture estimation unit 14 estimates, as the driver's reference posture, the most frequent detected driver posture among the detected driver posture A and the detected driver posture B during the reference setting period, which is detected most frequently by the feature detection unit 13.

[0086] In addition, the reference attitude update determination unit 17 determines whether or not to update the first reference attitude stored in the reference attitude memory unit 15 by comparing the difference between the first reference attitude stored in the reference attitude memory unit 15 and the second reference attitude stored in the reference attitude candidate memory unit 16 with a second threshold value.

[0087] With this configuration, the occupant state determination device 1 updates the reference posture when the driver changes his / her posture intentionally or due to habit due to long driving or changes in the driving environment (changes in weather or time of day). Therefore, the amount of change from the reference posture used to determine whether or not the posture is poor can be calculated with high accuracy. As a result, it is possible to suppress non-detection of posture imbalance that is determined not to be a posture imbalance even though the posture imbalance has occurred because the amount of change cannot be calculated with high accuracy, and to improve the detection accuracy of the occupant state.

[0088] Furthermore, in situation A shown in Figure 4, if one attempts to update the reference attitude using Patent Document 1 as described above, even if the occupant is in the reference attitude, it may be determined that the occupant has changed attitude from the reference attitude, and the occupant's reference attitude may be updated. Therefore, it is difficult to accurately calculate the amount of change from the reference posture used to determine whether or not the posture is impaired. As a result, even if posture is impaired, the amount of change cannot be accurately calculated, and therefore posture impairment is not detected and is determined not to be posture impairment, resulting in a problem of reduced accuracy in detecting the driver's inability to drive.

[0089] On the other hand, the occupant state determination device 1 estimates as the driver's reference posture the most frequent detected driver posture among the detected driver posture A and the detected driver posture B during the reference setting period, which is the most frequent detected driver posture detected by the feature detection unit 13, by the reference posture estimation unit 14.

[0090] By adopting such a configuration, the detected driver posture corresponding to the detected driver posture A in situation A is excluded from the most frequent value calculation in the reference posture estimation unit 14, and it is possible to avoid only the detected driver postures including the detected driver posture B being subject to the most frequent value calculation in the reference posture estimation unit 14.

[0091] As a result, it is possible to suppress the occurrence of differences in at least one of the facial direction and head position between the first reference posture updated by the reference posture update judgment unit 17 and the actual posture of the driver, and it is possible to avoid the driver being judged to have changed posture from the reference posture and the driver's reference posture being updated even when the driver is in the reference posture.

[0092] Therefore, the amount of change from the reference posture used to determine whether or not the posture is out of alignment can be calculated with high accuracy. As a result, it is possible to suppress non-detection of posture out of alignment, which is determined not to be posture out of alignment because the amount of change cannot be calculated with high accuracy even though posture out of alignment has occurred, and to improve the detection accuracy of the occupant state.

[0093] In addition, in the above embodiment 1, the reference attitude update determination unit 17 determines whether to update the first reference attitude by comparing the difference between the first reference attitude stored in the reference attitude memory unit 15 and the second reference attitude stored in the reference attitude candidate memory unit 16 with the second threshold value, but the reference attitude update determination unit 17 may determine whether to update the first reference attitude by machine learning.

[0094] In addition, in the above embodiment 1, the most frequent detected driver posture, which is the detected driver posture that is most frequently detected by the feature detection unit 13, among the detected driver posture A and the detected driver posture B, is estimated as the driver's reference posture, but the detected driver posture that is detected by the feature detection unit 13 with a frequency equal to or greater than a predetermined threshold value, among the detected driver posture A and the detected driver posture B, may be estimated as the reference posture.

[0095] In the above embodiment 1, the vehicle occupant for which the occupant state determination device 1 determines whether or not the vehicle is in an abnormal posture is the driver of the vehicle, but this is merely an example. The occupant state determination device 1 can determine whether or not the vehicle occupant is in an abnormal posture of any vehicle occupant other than the driver.

[0096] In addition, in the above embodiment 1, the occupant state judgment device 1 is an in-vehicle device mounted on a vehicle, and the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, reference posture memory unit 15, reference posture candidate memory unit 16, reference posture update judgment unit 17, posture collapse judgment unit 18, abnormal posture judgment unit 19, and output control unit 20 are provided in the in-vehicle device. Without being limited to this, some of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, reference posture memory unit 15, reference posture candidate memory unit 16, reference posture update determination unit 17, posture collapse determination unit 18, abnormal posture determination unit 19, and output control unit 20 may be mounted on the in-vehicle device of the vehicle, and the others may be provided in a server connected to the in-vehicle device via a network, so that a system is formed by the in-vehicle device and the server. In addition, the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, reference posture memory unit 15, reference posture candidate memory unit 16, reference posture update determination unit 17, posture collapse determination unit 18, abnormal posture determination unit 19, and output control unit 20 may all be provided in the server. [Explanation of symbols]

[0097] 1 Occupant state determination device, 2 Imaging device, 3 Output device, 11 Image acquisition unit, 12 Face detection unit, 13 Feature detection unit, 131 Face direction detection unit, 132 Head position detection unit, 133 Detection reliability calculation unit, 14 Reference posture estimation unit, 15 Reference posture storage unit, 16 Reference posture candidate storage unit, 17 Reference posture update determination unit, 18 Posture collapse determination unit, 19 Abnormal posture determination unit, 20 Output control unit, 100 Occupant state determination system, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory

Claims

1. a face detection unit that detects a face of a vehicle occupant in a captured image in which the face of the vehicle occupant is captured; a feature detection unit that detects a detected occupant posture, which is at least one of a facial direction and a head position of the occupant, by using the face of the occupant in the captured image detected by the face detection unit; a reference posture estimation unit that, when the detected occupant posture detected by the feature detection unit is less than a first threshold, determines the detected occupant posture as a reference posture which is at least one of a facial orientation and a head position serving as a reference for the occupant, and, when the detected occupant posture is equal to or greater than the first threshold, determines the detected occupant posture as a posture which the occupant can be considered to have changed from the reference posture, and estimates, as the reference posture, a mode which is the detected occupant posture detected most frequently by the feature detection unit among the detected occupant postures which the occupant can be considered to be the reference posture and the detected occupant postures which the occupant can be considered to have changed from the reference posture, which are acquired during a predetermined period; a reference orientation update determination unit that updates the first reference orientation to the second reference orientation when a difference between a first reference orientation, which is the reference orientation estimated using the most frequent value in the captured image, and a second reference orientation, which is the reference orientation estimated using the most frequent value in a captured image different from the captured image, is equal to or greater than a second threshold, and does not update the first reference orientation when the difference is less than the second threshold; a posture imbalance determination unit that performs a posture imbalance type determination as to whether the posture of the occupant corresponds to any one of a plurality of posture imbalance types by comparing the detected occupant posture with the first reference posture; an abnormal posture determination unit that determines that the occupant has an abnormal posture when the posture determination unit determines that the posture of the occupant falls under any of a plurality of posture deviation types, and determines that the occupant does not have an abnormal posture when the posture determination unit determines that the posture of the occupant does not fall under any of a plurality of posture deviation types; An occupant state determination device comprising:

2. the facial orientation of the occupant is represented by at least one of a yaw angle, which is a rotation angle about an axis in a vertical direction of the vehicle, a pitch angle, which is a rotation angle about an axis in a horizontal direction of the vehicle, and a roll angle, which is a rotation angle about an axis in a front-rear direction of the vehicle, which are predetermined references; The head position of the occupant is obtained by converting a point arbitrarily set on the face of the occupant in the captured image detected by the face detection unit into a point in real space. The occupant condition determination device according to claim 1 .

3. the reference attitude update determination unit determines, by machine learning, whether to update the first reference attitude to the second reference attitude or not to update the first reference attitude. The occupant condition determination device according to claim 1 or 2.

4. 3. The occupant state determination device according to claim 1, further comprising an output control unit that outputs warning information for warning the occupant when the abnormal posture determination unit determines that the occupant is in the abnormal posture.

5. The occupant state determination device according to claim 1 or 2, An occupant state determination system including: an imaging device installed in the vehicle so as to be able to image at least a face of the occupant; and an output device mounted in the vehicle and outputting a warning that the posture of the occupant is the abnormal posture, The occupant state determination device determines whether or not the posture of the occupant is the abnormal posture by using the captured image captured by the imaging device, and when it is determined that the posture of the occupant is the abnormal posture, causes the output device to output a warning that the posture of the occupant is the abnormal posture. Occupant status determination system.

6. A face detection unit detects a face of a vehicle occupant in a captured image in which the face of the vehicle occupant is captured; a feature detection unit detecting a detected occupant posture, which is at least one of a facial direction and a head position of the occupant, by using the face of the occupant in the captured image detected by the face detection unit; a reference posture estimation unit estimating, when the detected occupant posture detected by the feature detection unit is less than a first threshold, a detected occupant posture that can be regarded as a reference posture which is at least one of a facial orientation and a head position serving as a reference for the occupant, and when the detected occupant posture is equal to or greater than the first threshold, a detected occupant posture that can be regarded as a posture in which the occupant has changed from the reference posture, and estimating, as the reference posture, a mode that is the detected occupant posture that has been detected most frequently by the feature detection unit among the detected occupant postures that can be regarded as the reference posture of the occupant and the detected occupant postures that can be regarded as a posture in which the occupant has changed from the reference posture, which are acquired during a predetermined period; a reference orientation update determination unit updating the first reference orientation to the second reference orientation when a difference between a first reference orientation, which is the reference orientation estimated using the most frequent value in the captured image, and a second reference orientation, which is the reference orientation estimated using the most frequent value in a captured image different from the captured image, is equal to or greater than a second threshold, and not updating the first reference orientation when the difference is less than the second threshold; a posture deviation determination unit performing a posture deviation type determination as to whether the posture of the occupant corresponds to any one of a plurality of posture deviation types by comparing the detected occupant posture with the first reference posture; a step of determining that the occupant is in an abnormal posture when the posture deviation determination unit determines that the posture of the occupant corresponds to any one of a plurality of posture deviation types, and determining that the occupant is not in an abnormal posture when the posture deviation determination unit determines that the posture of the occupant does not correspond to any one of a plurality of posture deviation types; An occupant state determination method comprising:

7. A face detection unit detects a face of a vehicle occupant in a captured image in which the face of the vehicle occupant is captured; a feature detection unit detecting a detected occupant posture, which is at least one of a facial direction and a head position of the occupant, by using the face of the occupant in the captured image detected by the face detection unit; a reference posture estimation unit estimating, when the detected occupant posture detected by the feature detection unit is less than a first threshold, a detected occupant posture that can be regarded as a reference posture which is at least one of a facial orientation and a head position serving as a reference for the occupant, and when the detected occupant posture is equal to or greater than the first threshold, a detected occupant posture that can be regarded as a posture in which the occupant has changed from the reference posture, and estimating, as the reference posture, a mode that is the detected occupant posture that has been detected most frequently by the feature detection unit among the detected occupant postures that can be regarded as the reference posture of the occupant and the detected occupant postures that can be regarded as a posture in which the occupant has changed from the reference posture, which are acquired during a predetermined period; a reference orientation update determination unit updating the first reference orientation to the second reference orientation when a difference between a first reference orientation, which is the reference orientation estimated using the most frequent value in the captured image, and a second reference orientation, which is the reference orientation estimated using the most frequent value in a captured image different from the captured image, is equal to or greater than a second threshold, and not updating the first reference orientation when the difference is less than the second threshold; a posture deviation determination unit performing a posture deviation type determination as to whether the posture of the occupant corresponds to any one of a plurality of posture deviation types by comparing the detected occupant posture with the first reference posture; The abnormal posture determination unit A program that causes a computer to execute the steps of: determining that the occupant is in an abnormal posture if the posture collapse judgment unit determines that the posture of the occupant falls under any of a plurality of posture collapse types; and determining that the occupant is not in an abnormal posture if the posture collapse judgment unit determines that the posture of the occupant does not fall under any of a plurality of posture collapse types.

Images