Occupant state determination device, occupant state determination system, occupant state determination method, and occupant state determination program

Abstract

An occupant state determination device (1) according to the present disclosure comprises: a face detection unit (12) for detecting the face of an occupant in a captured image that includes a first range (6) being the range in which the face of a driver can exist and a second range (7) being the range in which the face of an occupant other than the driver can exist; a first face detection determination unit (51) for performing a first face detection determination in the captured image; a second face detection determination unit (52) for performing a second face detection determination by using a second captured image captured prior to the first captured image in which the face of the occupant was detected by the first face detection determination unit (51); an occupant determination unit (53) for performing an occupant determination that, in the case where a first face being the face of the occupant detected in the first captured image is located in the second range, the first face is the face of an occupant other than the driver; an abnormal posture determination unit (54) for performing an abnormal posture determination by using a third captured image which was captured prior to the first captured image and from which the face of the occupant was detected by the first face detection determination unit; and an abnormal state determination unit (55) for performing an abnormal state determination.

Description

Occupant state determination device, occupant state determination system, occupant state determination method, and occupant state determination program

[0001] The present disclosure relates to an occupant state determination device, an occupant state determination system, an occupant state determination method, and an occupant state determination program.

[0002] A technology has been developed that captures an image of the interior of a vehicle and uses information extracted from the occupant in the captured image to detect whether the occupant is in an abnormal state.

[0003] For example, in the abnormality determination device of Patent Document 1, when a person disappears from an operable range that is preset as a range within which the steering of a vehicle can be operated and then reappears within the operable range, if the appearance position, where the person appeared within the operable range, and the disappearance position, where the person disappeared from the operable range, are not the same, it is determined that the person who appeared and the person who disappeared are not the same person, and the driver is in an abnormal state.Furthermore, if the appearance position and the disappearance position are the same, it is determined that the person who appeared and the person who disappeared are the same person, and the driver is not in an abnormal state.

[0004] JP 2020-009179 A

[0005] However, there are various possible ways in which a person may disappear or appear in the operable range. Therefore, there may be an abnormal state of the driver that cannot be detected by a determination using the appearance position and disappearance position. For example, if the driver becomes abnormal and disappears from the operable range, and an occupant other than the driver reappears in the operable range from the same position as the driver's disappearance position in an attempt to assist the driver, the abnormality determination device of Patent Document 1 determines that the driver is not in an abnormal state. Therefore, the abnormality determination device of Patent Document 1, which determines the driver's abnormal state based on a comparison between the appearance position and the disappearance position as described above, has a problem in that the accuracy of detecting the occupant's state is reduced because the abnormal state of the driver is not sufficiently determined.

[0006] The present disclosure has been made to solve such problems, and has an object to provide an occupant state determination device that improves the accuracy of detecting the occupant state.

[0007] The occupant state determination device according to the present disclosure includes a face detection unit that detects the face of an occupant in a plurality of captured images each including both a first range, which is a range in which the face of the driver of a vehicle occupant may be present, and a second range, which is a range in which faces of persons other than the driver may be present; a first face detection determination unit that performs a first face detection determination as to whether the face of the occupant has been detected in the first range in the plurality of captured images; a second face detection determination unit that, when a determination result indicating that the face of the occupant has been detected in a first captured image of the plurality of captured images is obtained from the first face detection determination unit, performs a second face detection determination as to whether the face of the occupant has been detected in the first range using a second captured image of the plurality of captured images that was captured before the first captured image; and a determination unit that indicates that the face of the occupant has not been detected from the second face detection determination unit. The system is equipped with an occupant determination unit that, when a result is obtained, determines that a first face, which is the face of an occupant detected in a first captured image, is a face other than that of the driver if the first face is located within a second range of the first captured image, and that the first face is the face of the driver if the first face is not located within the second range of the first captured image, an abnormal attitude determination unit that, when a determination result indicating that the first face is a face other than that of the driver is obtained from the occupant determination unit, determines whether or not the driver is in an abnormal attitude using a third captured image among the multiple captured images that was captured before the first captured image and for which a determination result indicating that the face of an occupant was detected was output by the first face detection determination unit, and an abnormal state determination unit that determines whether or not the driver is in an abnormal state using the determination result of the abnormal attitude determination unit.

[0008] According to the occupant state determination device according to the embodiment of the present disclosure, it is possible to improve the accuracy of detecting the occupant state.

[0009] 1 is a block diagram showing the configuration of an occupant state determination device according to embodiment 1. FIG. 2 is a diagram showing the interior of a vehicle, a first range, and a second range according to embodiment 1. FIG. 3 is a diagram for explaining a plurality of posture imbalance types. FIG. 4 is a flowchart showing an example of the operation of the occupant state determination device according to embodiment 1. FIG. 5 is a diagram showing an example of the hardware configuration of the occupant state determination device according to embodiment 1.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1 An occupant state determination device 1 according to the present embodiment determines whether the posture of a vehicle occupant is abnormal based on an image of the vehicle occupant. In the following embodiment, as an example, the vehicle occupant whose posture is to be determined by the occupant state determination device 1 as being abnormal is the vehicle driver.

[0011] FIG. 1 is a block diagram showing the configuration of an occupant state determination device 1 according to this embodiment. FIG. 2 is a diagram showing the interior of a vehicle, a first range (described later), and a second range (described later). In this embodiment, it is assumed that the occupant state determination device 1 is mounted on a vehicle. However, the occupant state determination device 1 is not limited to being permanently installed in a vehicle, and may be implemented in a portable device that can be carried into a vehicle, such as a portable navigation device. Furthermore, a part or all of the occupant state determination device 1 may be implemented on a server capable of communicating with the vehicle.

[0012] The occupant state determination device 1 is connected to an image capture device 2 and an output device 3 , and the occupant state determination device 1 , the image capture device 2 and the output device 3 constitute an occupant state determination system 100 .

[0013] The imaging device 2 is mounted on the vehicle and is installed so as to be able to capture an imaging range that includes both a first range 6, which is a range in which the driver's face may be present, and a second range 7, which is a range in which faces other than the driver's may be present. Here, the first range 6 is a range in which the face of a driver seated in the driver's seat to drive the vehicle is located. If the front left seat among the seats installed inside the vehicle is the driver's seat, the first range 6 is the range indicated by the dotted line in FIG. 2 and is set to include a headrest 61 of the driver's seat and a portion of the backrest 62 of the driver's seat. Note that this is merely an example, and the first range 6 may be appropriately set by an administrator of the occupant state determination device 1 or may be stored in advance in the occupant state determination device 1. The second range 7 will be described in detail later.

[0014] The imaging device 2 is assumed to be installed, for example, near the center of the instrument panel in the vehicle width direction or on the center console. For example, the imaging device 2 may be a device shared with a so-called DMS (Driver Monitoring System) installed for the purpose of monitoring the interior of the vehicle. The imaging device 2 is a visible light camera or an infrared camera. The imaging device 2 captures images of the interior of the vehicle at intervals of, for example, 30 to 60 fps (frames per second) and outputs the captured images to the image acquisition unit 11 of the occupant state determination device 1. When the imaging device 2 captures images of the interior of the vehicle at intervals of 30 fps, it captures 30 frames per second.

[0015] The occupant state determination device 1 determines whether the driver is in an abnormal state using the captured image captured by the imaging device 2. For example, the occupant state determination device 1 determines whether the driver is in an abnormal state for each frame. Details of the occupant state determination device 1 will be described later. When the occupant state determination device 1 determines that the driver is in an abnormal state, it causes the output device 3 to output a warning that the driver is in an abnormal state.

[0016] The output device 3 is mounted on the vehicle and outputs a warning that the driver is in an abnormal state. The output device 3 is, for example, an audio output device such as a speaker. The output device 3 may be provided, for example, in an audio device provided in the vehicle. For example, when information for outputting a warning (hereinafter referred to as "warning information") is output from the occupant state determination device 1, the output device 3 outputs a warning sound or a voice message notifying the driver that the driver is in an abnormal state. Furthermore, the output device 3 may be, for example, a display device. For example, when warning information is output from the occupant state determination device 1, the output device 3 displays a message notifying the driver that the driver is in an abnormal state.

[0017] Furthermore, the output device 3 may be, for example, a turn signal, a hazard lamp, or a headlight provided on the exterior of the vehicle so as to be visible from other vehicles. For example, the occupant state determination device 1 may output warning information to the output device 3 to notify people outside the vehicle, such as occupants of the other vehicle, that the driver of the vehicle is in an abnormal state. The output device 3 may be mounted on the other vehicle, for example. The occupant state determination device 1 may transmit warning information to the output device 3 mounted on the other vehicle to output a voice or display a message to notify occupants of the other vehicle that the driver of the vehicle is in an abnormal state.

[0018] 1 , the occupant state determination device 1 includes an image acquisition unit 11, a face detection unit 12, a feature detection unit 13, a reference feature estimation unit 14, and a determination unit 1515. The feature detection unit 13 includes a face direction detection unit 31, a head position detection unit 32, a facial feature position detection unit 33, and a detection reliability calculation unit 34. The determination unit 15 includes a first face detection determination unit 51, a second face detection determination unit 52, an occupant determination unit 53, an abnormal posture determination unit 54, and an abnormal state determination unit 55.

[0019] The image acquisition unit 11 acquires a captured image from the imaging device 2. The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12.

[0020] The face detection unit 12 detects the driver's face in the first area 6 of the captured image acquired by the image acquisition unit 11. Specifically, the face detection unit 12 detects the driver's facial feature points, which indicate the driver's facial features, using a known image recognition technique, such as edge detection, on the captured image. The facial features include the corners and corners of the eyes, the nose, the mouth, the eyebrows, and the chin. The driver's facial feature points are represented, for example, by coordinates on the captured image. The face detection unit 12 also detects the driver's facial region. The driver's facial region is, for example, the smallest rectangle that encloses the outline of the driver's face. The driver's facial region is represented, for example, by the coordinates of the four corners of the smallest rectangle on the captured image. The face detection unit 12 outputs information on the detected driver's facial feature points and facial region (hereinafter referred to as "facial information") to the feature amount detection unit 13. The facial information is, for example, a captured image to which information capable of identifying the driver's facial feature points and facial region is added.

[0021] The feature amount detection unit 13 detects the feature amounts of the driver's face using the driver's face detected by the face detection unit 12. The feature amounts are the driver's facial direction, the driver's head position, and the driver's facial features. In this embodiment, the driver's facial direction and head position detected by the feature amount detection unit 13 are, for example, the facial direction and head position in real space. In this embodiment, the positions of the driver's facial features detected by the feature amount detection unit 13 are, for example, the positions of the facial features in a captured image. In this embodiment, as an example, the driver's facial features are the driver's eyes. In the following description, it is assumed that the driver's facial features are the driver's eyes.

[0022] The face direction detection unit 31 detects the face direction of the driver in real space based on the face information output from the face detection unit 12, specifically, based on the driver's face in the captured image detected by the face detection unit 12. The face direction detection unit 31 may detect the face direction of the driver using, for example, a known face direction detection technique for detecting the face direction from a captured image. The face direction of the driver is represented, for example, by at least one of a yaw angle, which is a rotation angle about an axis in the up-down direction of the vehicle, a pitch angle, which is a rotation angle about an axis in the left-right direction of the vehicle, and a roll angle, which is a rotation angle about an axis in the front-rear direction of the vehicle, which are predetermined references. The face direction detection unit 31 outputs information regarding the face direction of the driver (hereinafter referred to as "face direction information") to the reference feature estimation unit 14 and the determination unit 15. The face direction information includes at least one of information on the yaw angle, the pitch angle, and the roll angle.

[0023] The head position detection unit 32 detects the head position of the driver in real space based on the face information output from the face detection unit 12, more specifically, based on the face of the driver in the captured image detected by the face detection unit 12. In this embodiment, the head position of the driver in the captured image is indicated, for example, by the center of the driver's eyebrows. The head position detection unit 32 detects, for example, a point in real space corresponding to the center of the driver's eyebrows in the captured image as the driver's head position. Note that this is merely an example, and the position of the driver's head in the captured image may also be indicated, for example, by the center of the driver's facial area or the center of a line connecting the inner corners of the driver's eyes. In this case, the head position detection unit 32 detects, for example, a point in real space corresponding to the center of the driver's facial area in the captured image or the center of a line connecting the inner corners of the driver's eyes as the driver's head position in real space.

[0024] The head position detection unit 32 may detect the driver's head position using, for example, a known coordinate transformation technique that transforms points on a captured image into points in real space. The driver's head position is represented, for example, by coordinates in real space. The head position detection unit 32 outputs information related to the detected driver's head position (hereinafter referred to as "head position information") to the reference feature estimation unit 14 and the determination unit 15. The head position information includes coordinate information of the driver's head position.

[0025] The feature position detection unit 33 detects the position of the driver's eyes in the captured image based on the face information output from the face detection unit 12, specifically, based on the driver's face in the captured image detected by the face detection unit 12. The face detection unit 12 detects the positions of the driver's right eye and left eye in the captured image. In this embodiment, the position of the driver's eyes is defined as the center of a line connecting the inner corner and outer corner of the driver's eye in the captured image. The feature position detection unit 33 may detect the position of the driver's eyes using, for example, a known image recognition technique. The position of the driver's eyes is represented, for example, by coordinates on the captured image. The feature position detection unit 33 outputs information regarding the detected position of the driver's eyes (hereinafter referred to as "feature position information") to the determination unit 15. The feature position information includes coordinate information of the driver's right eye and coordinate information of the driver's left eye.

[0026] The detection reliability calculation unit 34 calculates face detection reliability indicating the reliability that at least one of the detected face direction and head position of the driver is at least one of a predetermined face direction and head position of the driver. The detection reliability calculation unit 34 also calculates face part detection reliability indicating the reliability that the detected face parts are at least one of the predetermined face parts. The detection reliability calculation unit 34 then outputs the calculated face detection reliability and face part detection reliability to the determination unit 15. The face detection reliability and face part detection reliability can be calculated, for example, using the methods described in the following documents. In the following document, the face detection reliability is calculated from the degree of match between at least one of the face direction and head position learned in advance from a large number of face images and at least one of the face direction and head position parts extracted from an image of the driver. The face part detection reliability is also calculated from the degree of match between the face parts learned in advance from a large number of face images and the face parts extracted from an image of the driver. “Rapid Object Detection Boosted Cascade of Simple Features”, 2001, Paul Viola, etal.

[0027] In the following description, the feature amounts detected by the feature amount detection unit 13 refer to the driver's facial orientation, the driver's head position, and the driver's facial features. Of the feature amounts detected by the feature amount detection unit 13, at least one of the driver's facial orientation and head position is defined as a first feature amount, and at least one of the face detection reliability and facial feature detection reliability derived from the feature amounts is defined as a second feature amount.

[0028] The reference feature estimation unit 14 estimates the driver's reference feature as a reference feature using at least one of the facial direction information and the head position information. Specifically, the reference feature estimation unit 14 estimates at least one of the driver's reference facial direction and head position as a reference feature using at least one of the driver's facial direction and head position detected during a predetermined period (hereinafter referred to as the "reference setting period"). The reference setting period is, for example, a period from when the vehicle ignition is turned on until a predetermined time (e.g., 10 seconds) has elapsed. Note that this is merely an example, and the reference setting period may be, for example, a predetermined time (e.g., 3 seconds) when the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km / h) and the steering angle is within a predetermined range (e.g., ±20 degrees).

[0029] Furthermore, while at least one of the driver's face direction and head position is used to estimate the reference feature, at least one of the driver's face direction and head position may be used to estimate the reference feature together with face detection reliability. For example, the reference feature estimation unit 14 estimates the average value of at least one of the driver's face direction and head position during a reference setting period as at least one of the face direction and head position in the driver's reference feature. The reference feature estimation unit 14 outputs information on the estimated driver's reference feature (hereinafter referred to as "reference feature information") to the determination unit 15. The reference feature information includes at least one of the face direction and head position in the driver's reference feature. The face direction in the driver's reference feature is expressed by at least one of the yaw angle, pitch angle, and roll angle. The face direction and head position in the driver's reference feature are expressed, for example, by coordinates in real space. Furthermore, the reference feature information includes coordinate information of the driver's head position as information on the driver's head position.

[0030] In the following description, the most recent captured image among the captured images acquired by the image acquisition unit 11 will be referred to as a first captured image. The first captured image may also be referred to as a current frame.

[0031] The first face detection determination unit 51 performs a first face detection determination as to whether or not the face of an occupant has been detected within the first range 6, using the second feature amount calculated in the captured image by the feature amount detection unit 13. The following description will discuss processing by the occupant state determination device when the captured image used for the first face detection determination is the first captured image. For example, when the first face detection determination unit 51 performs the first face detection determination in the first captured image, the first face detection determination unit 51 performs the first face detection determination as to whether or not the face of an occupant has been detected within the first range 6, using the first face detection feature amount, which is the second feature amount calculated in the first captured image by the feature amount detection unit 13.

[0032] First, the first face detection determination using the first face detection reliability, which is the first face detection feature, will be described. The first face detection determination unit 51 determines whether the first face detection reliability is equal to or greater than a fourth threshold. If the first face detection reliability is equal to or greater than the fourth threshold, the first face detection determination unit 51 determines that the occupant's face has been detected. If the first face detection reliability is less than the fourth threshold, the first face detection determination unit 51 determines that the occupant's face has not been detected.

[0033] When the first face detection reliability is equal to or greater than the fourth threshold, the first face detection determination unit 51 outputs a first determination result, which is a determination result including information indicating that the occupant's face was detected and information indicating the first captured image, to the second face detection determination unit 52, the occupant determination unit 53, and the abnormal posture determination unit 54. When the first face detection reliability is less than the fourth threshold, the first face detection determination unit 51 outputs a determination result including information indicating that the occupant's face was not detected and information indicating the first captured image to the abnormal posture determination unit 54.

[0034] Next, a first face detection determination will be described when using the first facial part detection reliability, which is the first face detection feature. The first face detection determination unit 51 determines whether the first facial part detection reliability is equal to or greater than a fifth threshold. If the first facial part detection reliability is equal to or greater than the fifth threshold, the first face detection determination unit 51 determines that the occupant's face has been detected. If the first facial part detection reliability is less than the fifth threshold, the first face detection determination unit 51 determines that the occupant's face has not been detected.

[0035] When the first facial part detection reliability is equal to or greater than the fifth threshold, the first face detection determination unit 51 outputs a determination result including information indicating that the occupant's face was detected and information indicating the first captured image to the second face detection determination unit 52. When the first facial part detection reliability is less than the fifth threshold, the first face detection determination unit 51 outputs a determination result including information indicating that the occupant's face was not detected and information indicating the first captured image to the abnormal posture determination unit 54.

[0036] Although the example in which the first face detection determination is made using either the first face detection reliability or the first facial part detection reliability has been shown, the present invention is not limited to this, and the first face detection determination may be made using the first face detection reliability and the first facial part detection reliability. In this case, the first face detection determination unit 51 determines that the occupant's face has been detected if the first face detection reliability is equal to or greater than a fourth threshold and the first facial part detection reliability is equal to or greater than a fifth threshold, and determines that the occupant's face has not been detected if the first face detection reliability is less than the fourth threshold or the first facial part detection reliability is less than the fifth threshold.

[0037] When the second face detection determination unit 52 receives a determination result from the first face detection determination unit 51 indicating that the occupant's face has been detected in the first captured image, the second face detection determination unit 52 performs a second face detection determination to determine whether the occupant's face has been detected in the first range 6, using a second captured image captured before the first captured image. Specifically, the second face detection determination unit 52 performs the second face detection determination using second face detection features, which are second features calculated in the second captured image by the feature detection unit 13. Specifically, the second captured image is an image captured one frame before the first captured image. The second captured image may also be referred to as the immediately preceding frame.

[0038] First, the second face detection determination using the second face detection reliability, which is the second face detection feature, will be described. The second face detection determination unit 52 determines whether the second face detection reliability is equal to or greater than a fourth threshold. If the second face detection reliability is equal to or greater than the fourth threshold, the second face detection determination unit 52 determines that the occupant's face has been detected. If the second face detection reliability is less than the fourth threshold, the second face detection determination unit 52 determines that the occupant's face has not been detected.

[0039] When the second face detection reliability is equal to or greater than the fourth threshold, the second face detection determination unit 52 outputs a determination result including information indicating that the occupant's face was detected and information indicating the second captured image to the abnormal posture determination unit 54. When the second face detection reliability is less than the fourth threshold, the second face detection determination unit 52 outputs a determination result including information indicating that the occupant's face was not detected and information indicating the second captured image to the occupant determination unit 53.

[0040] Next, a second face detection determination will be described when using the second facial part detection reliability, which is the second face detection feature. The second face detection determination unit 52 determines whether the second facial part detection reliability is equal to or greater than a fifth threshold. If the second facial part detection reliability is equal to or greater than the fifth threshold, the second face detection determination unit 52 determines that the occupant's face has been detected. If the second facial part detection reliability is less than the fifth threshold, the second face detection determination unit 52 determines that the occupant's face has not been detected.

[0041] When the second facial part detection reliability is equal to or greater than the fifth threshold, the second face detection determination unit 52 outputs a determination result including information indicating that the occupant's face was detected and information indicating the second captured image to the abnormal posture determination unit 54. When the second facial part detection reliability is less than the fifth threshold, the second face detection determination unit 52 outputs a determination result including information indicating that the occupant's face was not detected and information indicating the second captured image to the occupant determination unit 53.

[0042] Although the example in which the second face detection determination is performed using either the second face detection reliability or the second facial part detection reliability has been shown, the second face detection determination may be performed using both the second face detection reliability and the second facial part detection reliability. In this case, the second face detection determination unit 52 determines that the occupant's face has been detected if the second face detection reliability is equal to or greater than a fourth threshold and the second facial part detection reliability is equal to or greater than a fifth threshold, and determines that the occupant's face has not been detected if the second face detection reliability is less than the fourth threshold or the second facial part detection reliability is less than the fifth threshold.

[0043] When the occupant determination unit 53 obtains a determination result from the second face detection determination unit 52 indicating that the occupant's face could not be detected, if the first face, which is the occupant's face detected in the first captured image, is located within a second range of the first captured image, it determines that the first face is a face other than that of the driver, and if the first face is not located within the second range of the first captured image, it determines that the first face is the face of the driver.

[0044] Here, the second range 7 will be described in the case where the face other than the driver is the face of another occupant other than the driver. The second range 7 is a range where the face of the other occupant is considered to be located. More specifically, the second range 7 includes a range where the faces of other occupants seated in seats other than the driver's seat of the vehicle are located and a range where the faces of other occupants who attempt to assist the driver in an abnormal state are located. As shown in FIG. 2 , the second range 7 is indicated by a dashed line and is set to include a portion of the backrest 62 of the driver's seat, headrests of seats other than the driver's seat, such as the headrest 71 of the passenger seat and the headrest 73 of the rear seat, and portions of the backrests of seats other than the driver's seat, such as the backrest 72 of the passenger seat and the backrest 74 of the rear seat. Note that this is merely an example, and the second range 7 may be appropriately set by an administrator of the occupant state determination device 1 or may be stored in advance in the occupant state determination device 1.

[0045] In this embodiment, an example has been given in which the faces other than the driver are the faces of other passengers, but this is not limited to this, and faces other than the driver also include patterns with shapes or structural features similar to those of a human face, specifically faces printed on masks or clothing, landscape patterns, etc.

[0046] Here, the reason why the occupant determination unit 53 performs occupant determination will be explained. First, when the occupant determination unit 53 obtains a determination result from the second face detection determination unit 52 indicating that the occupant's face could not be detected (hereinafter referred to as "Case A"), this refers to a case where the occupant's face was detected in the first range 6 set in the first captured image but not in the first range 6 set in the second captured image. As described above, the second captured image was captured before the first captured image. In other words, Case A refers to a case where the occupant's face could no longer be detected by the occupant state determination device 1, but the occupant's face can be detected again (hereinafter referred to as "face lost return"). For example, the face lost return includes a state where the driver disappears from the first range 6 to pick up an object near the driver's seat and then reappears in the first range 6, a state where the driver becomes abnormal and disappears from the first range 6, and another occupant reappears in the first range 6 to assist the driver, etc. In other words, when face loss recovery occurs, there is a possibility that the face of an occupant detected in the first range 6 set in the first captured image is a face other than that of the driver. Therefore, if the occupant determination unit 53 does not perform occupant determination, an abnormal posture determination, which will be described later, is performed using information about faces other than that of the driver. In other words, if information about faces other than that of the driver, which are not the subject of the occupant state determination device 1's determination of whether or not an abnormal state exists, is used in the abnormal posture determination, the accuracy of the abnormal state determination, which will be described later, performed using the results of the abnormal posture determination, decreases. As a result, there is a possibility that the accuracy of detecting the occupant state in the occupant state determination device 1 decreases. For the above reasons, the occupant determination unit 53 performs occupant determination.

[0047] Specifically, the occupant determination unit 53 performs occupant determination by determining whether or not the face of the occupant is located at least in the second range 7. In this embodiment, five occupant determinations performed by the occupant determination unit 53 will be described. The occupant determination unit 53 performs one of the five occupant determinations.

[0048] In the following description, the first feature detected by the feature detection unit 13 in the first captured image will be referred to as the first judgment feature, and the first feature detected by the feature detection unit 13 in the third captured image, which is an image captured before the first captured image and for which the first face detection judgment unit 51 has output a judgment result indicating that the occupant's face has been detected, will be referred to as the third judgment feature.

[0049] Here, the third captured image will be described. The third captured image is determined using the determination result output by the first face detection determination unit 51 before the occupant determination using the first captured image. Specifically, the third captured image is an image captured one or more frames before the first captured image, and is an image in which the first face detection determination unit 51 determines that the occupant's face has been detected.

[0050] For example, a case will be described in which the first face detection determination unit 51 determines that the occupant's face was detected in the first captured image, determines that the occupant's face was not detected in the captured images one to three frames before the first captured image, and determines that the occupant's face was detected in the first captured image four frames before the first captured image (hereinafter referred to as "Case B"). In Case B, the first face detection determination unit 51 outputs a determination result corresponding to the captured image four frames before the first captured image to the occupant determination unit 53, and then outputs a determination result corresponding to the first captured image. Then, in Case A, the occupant determination unit 53 performs occupant determination using the captured image four frames before the first captured image as the third captured image.

[0051] In other words, Case B refers to a case where the occupant's face was detected in the image captured four frames before the first captured image, but was not detected in the images captured one to three frames before the first captured image, and was then able to be detected again in the first captured image. Here, a case where the occupant's face was previously detected but is now no longer detectable in the occupant state determination device 1 is referred to as a "face lost" occurrence, a case where the occupant's face can no longer be detected is referred to as "face lost," and, as described above, a case where the occupant's face becomes undetectable but is then redetectable is referred to as "face lost recovery." The first captured image is the image captured after face lost recovery, the second captured image is the image captured during face lost, and the third captured image is the image captured before face lost occurred. In the following description, the third captured image may also be referred to as a frame before face lost occurred.

[0052] First, the occupant determination performed using the first head position, which is the first determination feature, will be described. The occupant determination unit 53 determines whether the first head position falls within the second range 7. If the first head position falls within the second range 7, the occupant determination unit 53 determines that the first face is the face of a person other than the driver, and if the first head position does not fall within the second range 7, the occupant determination unit 53 determines that the first face is the face of the driver.

[0053] When the first head position falls within the second range 7, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54. When the first head position does not fall within the second range 7, the occupant determination unit 53 outputs a determination result including information indicating that the first face is the face of the driver to the abnormal posture determination unit 54.

[0054] Next, occupant determination performed using the first head position and first facial orientation, which are the first determination features, will be described. The occupant determination unit 53 determines whether the first head position falls within the second range 7 and whether the absolute value of the first facial orientation is less than a first threshold. If the first head position falls within the second range and the absolute value of the first facial orientation is less than the first threshold, the occupant determination unit determines that the first face is a face other than that of the driver. If the first head position does not fall within the second range or the absolute value of the first facial orientation is equal to or greater than the first threshold, the occupant determination unit determines that the first face is a face of the driver. Here, the third range may be appropriately set by an administrator of the occupant state determination device 1 or may be stored in advance in the occupant state determination device 1.

[0055] When the first head position falls within the second range 7 and the first facial orientation falls within the third range, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54. When the first head position does not fall within the second range 7 or the first facial orientation does not fall within the third range, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face of the driver to the abnormal posture determination unit 54.

[0056] Next, an occupant determination performed using the first head position, which is the first determination feature, and the second head position, which is the third determination feature, will be described. The occupant determination unit 53 determines whether the first head position falls within the second range 7 and whether the absolute value of a head position difference, which is the difference between the first head position and the third head position, is equal to or greater than a second threshold. If the first head position falls within the second range 7 and the absolute value of the head position difference is equal to or greater than the second threshold, the occupant determination unit 53 determines that the first face is a face other than that of the driver. If the first head position does not fall within the second range 7 or the absolute value of the head position difference is less than the second threshold, the occupant determination unit 53 determines that the first face is a face of the driver. Here, the second threshold may be appropriately set by an administrator of the occupant state determination device 1 or may be stored in advance in the occupant state determination device 1.

[0057] If the first head position falls within the second range 7 and the absolute value of the head position difference is equal to or greater than the second threshold, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54. If the first head position does not fall within the second range 7 or the absolute value of the head position difference is less than the second threshold, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face of the driver to the abnormal posture determination unit 54.

[0058] Next, an occupant determination performed using the first determination feature values, i.e., the first head position and the first facial direction, and the third determination feature value, i.e., the second facial direction, will be described. The occupant determination unit 53 determines whether the first head position falls within the second range 7 and whether the absolute value of a facial direction difference, which is the difference between the first facial direction and the third facial direction, is equal to or greater than a third threshold. If the first head position falls within the second range 7 and the absolute value of the facial direction difference is equal to or greater than the third threshold, the occupant determination unit 53 determines that the first face is a face other than that of the driver. If the first head position does not fall within the second range 7 or the absolute value of the facial direction difference is less than the third threshold, the occupant determination unit 53 determines that the first face is a face of the driver. Here, the third threshold may be appropriately set by an administrator of the occupant state determination device 1 or may be stored in advance in the occupant state determination device 1.

[0059] If the first head position falls within the second range 7 and the absolute value of the facial direction difference is equal to or greater than the third threshold, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54. If the first head position does not fall within the second range 7 or the absolute value of the facial direction difference is less than the third threshold, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face of the driver to the abnormal posture determination unit 54.

[0060] Next, occupant determination will be described, which is performed using the first head position, which is the first determination feature, and recognition information acquired from a recognition unit (not shown) configured to recognize a person in the first captured image. The occupant determination unit 53 determines whether the first recognition information, which is information for recognizing a first face, matches driver recognition information, which is information for recognizing the face of the driver. If the first head position falls within the second range 7 and the first recognition information does not match the driver recognition information, the occupant determination unit 53 determines that the first face is the face of a person other than the driver. If the first head position does not fall within the second range 7 or the first recognition information matches the driver recognition information, the occupant determination unit 53 determines that the first face is the face of the driver.

[0061] When the first head position falls within the second range 7 and the first recognition information does not match the driver identification information, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54. When the first head position does not fall within the second range 7 or the first recognition information matches the driver identification information, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face of the driver to the abnormal posture determination unit 54.

[0062] The abnormal attitude determination unit 54 performs abnormal attitude determination using the first captured image when it acquires a determination result indicating that an occupant's face has been detected from the second face detection determination unit 52, or when it acquires a determination result indicating that the first face is the face of the driver from the occupant determination unit 53. The abnormal attitude determination unit 54 performs abnormal attitude determination using the third captured image when it acquires a determination result indicating that an occupant's face has not been detected from the first face detection determination unit 51, or when it acquires a determination result indicating that the first face is the face of a face other than the driver from the occupant determination unit 53.

[0063] Specifically, the abnormal posture determination unit 54 performs abnormal posture determination using the first determination feature when it acquires a determination result indicating that an occupant's face has been detected from the second face detection determination unit 52, or when it acquires a determination result indicating that the first face is the face of the driver from the occupant determination unit 53. The abnormal posture determination unit 54 performs abnormal posture determination using the third determination feature when it acquires a determination result indicating that an occupant's face has not been detected from the first face detection determination unit 51, or when it acquires a determination result indicating that the first face is the face of a face other than that of the driver from the occupant determination unit 53.

[0064] Here, we will explain why the abnormal posture judgment unit 54 performs abnormal posture judgment using the first judgment feature when it obtains a judgment result from the second face detection judgment unit 52 indicating that an occupant's face has been detected, or when it obtains a judgment result from the occupant judgment unit 53 indicating that the first face is the driver's face.

[0065] First, when the abnormal posture determination unit 54 obtains a determination result from the second face detection determination unit 52 indicating that the occupant's face has been detected (hereinafter referred to as "Case C"), this refers to a case where the occupant's face is detected in the first range 6 set in the first captured image and also in the first range 6 set in the second captured image. In other words, Case C refers to a state where the occupant's face is continuously detected by the occupant state determination device 1. Therefore, the occupant's face detected in the first range 6 set in the first captured image is considered to be the driver's face. Therefore, the abnormal posture determination unit 54 performs abnormal posture determination using the first determination feature.

[0066] Next, when the abnormal posture determination unit 54 obtains a determination result from the occupant determination unit 53 indicating that the first face is the driver's face, this refers to a case where the occupant's face is detected again after face loss recovery. Therefore, the occupant's face detected in the first range 6 set in the first captured image is considered to be the driver's face. Therefore, the abnormal posture determination unit 54 performs abnormal posture determination using the first determination feature.

[0067] Next, we will explain why the abnormal posture judgment unit 54 performs abnormal posture judgment using the third judgment feature when it obtains a judgment result from the first face detection judgment unit 51 indicating that the occupant's face could not be detected, or when it obtains a judgment result from the occupant judgment unit 53 indicating that the first face is the face of a person other than the driver.

[0068] First, when the abnormal posture determination unit 54 obtains a determination result from the first face detection determination unit 51 indicating that the occupant's face could not be detected, this refers to a case where face loss has occurred or the face has been lost. In other words, since the occupant's face cannot be detected in the first range 6 set in the first captured image, the abnormal posture determination unit 54 performs abnormal posture determination using information about the occupant's face detected in the first range 6 set in the third captured image. Therefore, the abnormal posture determination unit 54 performs abnormal posture determination using the third determination feature amount.

[0069] Next, when the abnormal posture determination unit 54 obtains a determination result from the occupant determination unit 53 indicating that the first face is a face other than that of the driver, this refers to a case where the face of the occupant detected again is a face other than that of the driver after face loss recovery. Therefore, the occupant's face detected in the first range 6 set in the first captured image is considered to be a face other than that of the driver. Furthermore, because the occupant's face cannot be detected in the first range 6 set in the second captured image, the abnormal posture determination unit 54 performs abnormal posture determination using information about the occupant's face detected in the first range 6 set in the third captured image. Therefore, the abnormal posture determination unit 54 performs abnormal posture determination using the third determination feature.

[0070] The abnormal posture determination unit 54 will be described in detail below. Based on the determination results obtained from the first face detection determination unit 51, the second face detection determination unit 52, or the occupant determination unit 53, the abnormal posture determination unit 54 determines whether to use the first captured image or the third captured image as a determination-use captured image, which is an image used for abnormal posture determination. The abnormal state determination unit 55 then uses the first feature quantity detected in the captured image used as the determination-use captured image as a determination feature quantity. The abnormal posture determination unit 54 then compares the determination feature quantity with the reference feature quantity obtained from the reference feature quantity estimation unit 14 to determine whether the driver's posture corresponds to one of a plurality of predetermined posture imbalance types (hereinafter referred to as "posture imbalance types"). If the driver's posture corresponds to one of the posture imbalance types, the abnormal state determination unit 55 determines that the occupant has an abnormal posture. If the driver's posture does not correspond to one of the posture imbalance types, the abnormal state determination unit 55 determines that the occupant does not have an abnormal posture.

[0071] FIG. 3 (partially modified from the "Basic Design Document for Driver Abnormality Automatic Detection System" by the Advanced Safety Vehicle Promotion Study Group, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, March 2018) is a diagram for explaining multiple types of poor posture. As shown in FIG. 3, there are multiple types of poor posture. FIG. 3 illustrates the following: "head down" in which the driver leans forward and continues to have their face close to the steering wheel; "head down" in which the driver continues to have their face facing down; "backward lean" in which the driver's upper body leans backward and continues to have their face facing up; "backward lean" in which the driver's upper body leans up and continues to have their face facing up; "head tilt" in which the driver's face tilts to the left or right; "sideways lean" in which the driver's upper body leans to the left or right and their face tilts in the same direction; and "leaning to the side" in which the driver's upper body leans to the left or right. In the present disclosure, the multiple pre-set posture abnormality types are "head down," "head down," "backward arch," "arched back," "head tilted to the side only," "side tilt," and "leaning to the side" shown in Fig. 3. Information on the multiple posture abnormality types is stored in a location that can be referenced by the abnormal posture determination unit 54.

[0072] Specifically, the abnormal posture determination unit 54 determines which of a plurality of posture abnormality types the driver belongs to depending on how much the determination feature has changed compared to the reference feature. Note that the conditions for determining which posture abnormality type the driver belongs to based on how much the determination feature has changed compared to the reference feature are set in advance and stored in the abnormal posture determination unit 54.

[0073] If the driver's posture corresponds to one of the posture deviation types, the abnormal posture determination unit 54 outputs a determination result including information indicating that the driver has an abnormal posture to the abnormal state determination unit 55, and if the driver's posture does not correspond to one of the posture deviation types, the abnormal posture determination unit 54 outputs a determination result including information indicating that the driver does not have an abnormal posture to the abnormal state determination unit 55.

[0074] The abnormal state determination unit 55 uses the determination result output by the abnormal posture determination unit 54 to determine whether the driver is in an abnormal state.

[0075] When the abnormal posture determination unit 54 determines that the driver has an abnormal posture, the abnormal state determination unit 55 determines the duration of the state in which the driver has an abnormal posture. For example, the abnormal state determination unit 55 associates the determination result acquired from the abnormal posture determination unit 54 with information on the date and time when the determination result was acquired and stores the result in a storage unit (not shown). The abnormal state determination unit 55 may determine the duration of the state in which the driver has an abnormal posture from the stored determination result. When the duration of the state in which the driver has an abnormal posture reaches a sixth threshold, the abnormal state determination unit 55 determines that the driver is in an abnormal state.

[0076] The abnormal state determination unit 55 outputs the determination result as to whether the driver is in an abnormal state to the output device 3 .

[0077] 4 is a flowchart showing an example of the operation of the occupant state determination device 1 according to this embodiment. The processing executed by the occupant state determination device 1 according to this embodiment will be described with reference to the flowchart shown in FIG. The operation of the occupant state determination device 1 is started, for example, after the ignition of the vehicle is turned on. It is also preferable to perform the occupant state determination processing every time an image is acquired from the imaging device 2, that is, for each frame.

[0078] The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST1). The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12.

[0079] After step ST1, the face detection unit 12 detects the driver's face in the captured image acquired by the image acquisition unit 11 in step ST1 (step ST2). The face detection unit 12 outputs face information to the feature amount detection unit 13.

[0080] After step ST2, the face direction detection unit 31 detects the face direction of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The face direction detection unit 31 outputs the face direction information to the reference feature estimation unit 14 and the determination unit 15.

[0081] After step ST2, the head position detection unit 32 detects the head position of the driver in real space based on the face information output from the face detection unit 12 in step ST2, more specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The head position detection unit 32 outputs the head position information to the reference feature estimation unit 14 and the determination unit 15.

[0082] After step ST2, the facial feature position detection unit 33 detects the positions of the facial features of the driver in real space based on the face information output from the face detection unit 12 in step ST2, more specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The facial feature position detection unit 33 outputs the facial feature position information to the determination unit 15.

[0083] After step ST3, the detection reliability calculation unit 34 calculates face detection reliability and facial part detection reliability based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST4). The detection reliability calculation unit 34 outputs the face detection reliability and the facial part detection reliability to the determination unit 15.

[0084] After step ST4, the reference feature estimation unit 14 estimates a reference feature using at least one of the face orientation information and the head position information (step ST5). The detection reliability calculation unit 34 outputs the reference feature to the determination unit 15.

[0085] After step ST5, the first face detection determination unit 51 performs a first face detection determination to determine whether or not the face of an occupant has been detected in the first range 6 using the first face detection feature amount (step ST6). Step ST6 indicates whether or not the first face detection determination unit 51 has succeeded in detecting the face of an occupant in the current frame.

[0086] If the first face detection determination unit 51 has detected the face of the occupant (if "YES" in step ST6), the second face detection determination unit 52 uses the second captured image to perform a second face detection determination as to whether or not the face of the occupant has been detected in the first range 6 (step ST7). Step ST7 indicates whether or not the second face detection determination unit 52 has succeeded in detecting the face of the occupant in the immediately preceding frame.

[0087] If the second face detection determination unit 52 cannot detect the face of an occupant ("NO" in step ST7), the occupant determination unit 53 performs occupant determination as to whether the first face is a face other than that of the driver by using the second range 7 (step ST8). Step ST9 means that the occupant determination unit 53 determines whether the face of an occupant detected in the current frame is a face other than that of the driver.

[0088] If the first face is a face other than that of the driver (if "YES" in step ST8), the occupant determination unit 53 and the abnormal posture determination unit 54 use the frame before face loss occurred as the captured image for determination (step ST9).

[0089] If the second face detection judgment unit 52 is able to detect the face of an occupant (if "YES" in step ST7), or if the first face is the face of the driver (if "NO" in step ST8), the abnormal posture judgment unit 54 adopts the current frame as the captured image for judgment (step ST10).

[0090] If the first face detection judgment unit 51 is unable to detect the occupant's face (if the answer is "NO" in step ST6), the occupant judgment unit 53 and the abnormal posture judgment unit 54 use the frame before the face loss occurred as the captured image for judgment (step ST11).

[0091] After step ST9, step ST10, or step ST11, the abnormal posture determination unit 54 performs abnormal posture determination using the determination feature amount and the reference feature amount (step ST12).

[0092] If the abnormal posture determination unit 54 determines that the driver is in an abnormal posture (if "YES" in step ST12), the abnormal state determination unit 55 determines whether the duration of the driver's abnormal posture is greater than or equal to a sixth threshold value (step ST13).

[0093] If the duration of the state in which the driver is in an abnormal posture is equal to or longer than the sixth threshold value ("YES" in step ST13), the abnormal state determination unit 55 determines that the driver is in an abnormal state (step ST14).

[0094] If the abnormal posture determination unit 54 determines that the driver is not in an abnormal posture (if the answer is "NO" in step ST12), or if the duration of the driver's abnormal posture is less than the sixth threshold (if the answer is "NO" in step ST13), the abnormal state determination unit 55 determines that the driver is not in an abnormal state (step ST15).

[0095] 5 and 6 are diagrams showing an example of the hardware configuration of the occupant state determination device 1 according to the first embodiment. In the first embodiment, the functions of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference feature estimation unit 14, and the determination unit 15 are realized by a processing circuit 1001. That is, the occupant state determination device 1 includes the processing circuit 1001 for performing control to determine whether a vehicle occupant is in an abnormal state based on a captured image. The processing circuit 1001 may be dedicated hardware as shown in FIG. 5, or a processor 1004 that executes a program stored in a memory 1005 as shown in FIG. 6.

[0096] When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0097] When the processing circuit is the processor 1004, the functions of the image acquisition unit 11, the face detection unit 12, the feature amount detection unit 13, the reference feature amount estimation unit 14, and the determination unit 15 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 1005. The processor 1004 reads and executes the program stored in the memory 1005 to perform the functions of the image acquisition unit 11, the face detection unit 12, the feature amount detection unit 13, the reference feature amount estimation unit 14, and the determination unit 15. In other words, the occupant state determination device 1 includes the memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST15 of FIG. 4 described above. It can also be said that the programs stored in the memory 1005 cause the computer to execute the processing procedures or methods of the image acquisition unit 11, face detection unit 12, feature amount detection unit 13, reference feature amount estimation unit 14, and determination unit 15. Here, the memory 1005 corresponds to, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a minidisk, a DVD (Digital Versatile Disc), or the like.

[0098] It is also possible to realize some of the functions of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference feature estimation unit 14, the determination unit 15, and the output control unit 16 with dedicated hardware and some with software or firmware. For example, the function of the image acquisition unit 11 can be realized by a processing circuit 1001 as dedicated hardware, and the functions of the face detection unit 12, the feature detection unit 13, the reference feature estimation unit 14, and the determination unit 15 can be realized by the processor 1004 reading and executing programs stored in the memory 1005.

[0099] The storage unit (not shown) is, for example, a memory 1005. The occupant state determination device 1 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the imaging device 2 or the output device 3.

[0100] The occupant state determination device 1 of this embodiment includes a face detection unit 12 that detects the face of an occupant in a plurality of captured images that each include both a first range 6, which is a range in which the face of the driver of the vehicle may be present, and a second range 7, which is a range in which the face of an occupant other than the driver may be present; a first face detection determination unit 51 that performs a first face detection determination as to whether the face of the occupant has been detected in the first range 6 in the plurality of captured images; a second face detection determination unit 52 that, when a determination result indicating that the face of the occupant has been detected in a first captured image of the plurality of captured images is obtained from the first face detection determination unit 51, performs a second face detection determination as to whether the face of the occupant has been detected in the first range 6 using a second captured image of the plurality of captured images that was captured before the first captured image; and a determination result indicating that the face of the occupant has not been detected from the second face detection determination unit 52. an occupant determination unit that, when a determination result indicating that the first face is a face other than the driver's, determines that the first face is a face other than the driver's when a first face, which is the face of an occupant detected in the first captured image, is located in the second range 7 of the first captured image, and that the first face is a face of the driver when the first face is not located in the second range 7 of the first captured image; an abnormal attitude determination unit that, when a determination result indicating that the first face is a face other than the driver's, is obtained from the occupant determination unit 53, determines whether or not the driver is in an abnormal attitude using a third captured image, which was taken before the first captured image among the multiple captured images and for which a determination result indicating that the face of an occupant was detected was output by the first face detection determination unit 51; and an abnormal state determination unit that, using the determination result of the abnormal attitude determination unit 54, determines whether or not the driver is in an abnormal state. When the occupant's face disappears from the first range 6 and then reappears in the first range 6, the occupant state determination device 1 uses the second range 7 to determine whether the first face is a face other than that of the driver. With this configuration, even if the driver becomes abnormal and disappears from the first range 6, and another occupant attempts to assist the driver and reappears in the first range 6 from the same direction as the driver's disappearance, the occupant state determination device 1 can determine that the occupant's face that appears in the first range 6 is a face other than that of the driver. This makes it possible to avoid a situation in which information about the face of a person other than the driver, which is not the target of abnormal state determination, is used in abnormal posture determination.As a result, the occupant state determination device 1 can improve the accuracy of detecting the occupant state.

[0101] The occupant state determination system 100 of this embodiment includes the occupant state determination device 1 of this embodiment, an imaging device 2 installed in the vehicle so as to be able to capture an image of at least the face of the occupant, and an output device 3 mounted on the vehicle and configured to output a warning that the occupant is in an abnormal state. The occupant state determination device 1 determines whether the occupant is in an abnormal state using an image captured by the imaging device 2, and if it determines that the occupant is in an abnormal state, causes the output device 3 to output a warning that the occupant is in an abnormal state. When the occupant's face disappears from the first range 6 and reappears in the first range 6, the occupant state determination system 100 uses the second range 7 to determine whether the first face is a face other than that of the driver. With this configuration, even if the driver becomes abnormal and disappears from the first range 6, and another occupant reappears in the first range 6 from the same direction as the driver's disappearance in an attempt to assist the driver, it is possible to determine that the face of the occupant appearing in the first range 6 is a face other than that of the driver. Therefore, it is possible to avoid a situation in which information about the face of a person other than the driver, which is not the target of abnormal state determination, is used for abnormal posture determination, thereby improving the accuracy of detecting the occupant state in the occupant state determination system 100.

[0102] The occupant state determination method of this embodiment includes a face detection step in which the face detection unit 12 detects the face of an occupant in a plurality of captured images each including both a first range 6, which is a range in which the face of the driver of the vehicle occupants can be present, and a second range 7, which is a range in which faces other than the driver can be present; a first face detection determination step in which the first face detection determination unit 51 makes a first face detection determination as to whether the face of the occupant has been detected in the first range 6 in the plurality of captured images; a second face detection determination step in which the second face detection determination unit 52, when obtaining from the first face detection determination unit 51 a determination result indicating that the face of the occupant has been detected in a first captured image of the plurality of captured images, makes a second face detection determination as to whether the face of the occupant has been detected in the first range 6 using a second captured image of the plurality of captured images that was captured before the first captured image; and a second face detection determination step in which the occupant determination unit 53 determines whether the face of the occupant has been detected from the second face detection determination unit 52. an occupant determination step in which, when a determination result indicating that the first face was not located in the second range 7 of the first captured image is obtained, the abnormal posture determination unit 54 performs an occupant determination that the first face is a face other than that of the driver if the first face, which is the face of an occupant detected in the first captured image, is located in the second range 7 of the first captured image, and that the first face is the face of the driver if the first face is not located in the second range 7 of the first captured image; an abnormal posture determination step in which, when the abnormal posture determination unit 54 obtains a determination result indicating that the first face is a face other than that of the driver from the occupant determination unit 53, the abnormal posture determination unit 54 performs an abnormal posture determination as to whether or not the driver is in an abnormal posture using a third captured image among the multiple captured images that was taken before the first captured image and for which a determination result indicating that the face of an occupant was detected was output by the first face detection determination unit 51; and an abnormal state determination step in which the abnormal state determination unit 55 performs an abnormal state determination as to whether or not the driver is in an abnormal state using the determination result of the abnormal posture determination unit 54. In the occupant state determination method, when the occupant's face disappears from the first range 6 and then reappears in the first range 6, it is determined whether the first face is a face other than that of the driver by using the second range 7. With this configuration, even if the driver becomes abnormal and disappears from the first range 6, and another occupant attempts to assist the driver and reappears in the first range 6 from the same direction as the driver's disappearance, it is possible to determine that the occupant's face that appears in the first range 6 is a face other than that of the driver.Therefore, it is possible to avoid a situation in which information about the face of a person other than the driver, which is not the target of the abnormal state determination, is used for the abnormal posture determination, thereby improving the accuracy of detecting the occupant state in the occupant state determination method.

[0103] The occupant state determination program of the present embodiment includes a computer having a face detection function in which a face detection unit detects the face of an occupant in a plurality of captured images each including both a first range 6, which is a range in which the face of the driver of the vehicle occupants may be present, and a second range 7, which is a range in which the faces of persons other than the driver may be present; a first face detection determination function in which a first face detection determination unit 51 makes a first face detection determination as to whether or not the face of an occupant has been detected in the first range 6 in the plurality of captured images; a second face detection determination function in which a second face detection determination unit 52, when obtaining from the first face detection determination unit 51 a determination result indicating that the face of an occupant has been detected in a first captured image of the plurality of captured images, makes a second face detection determination as to whether or not the face of an occupant has been detected in the first range 6 using a second captured image of the plurality of captured images that was captured before the first captured image; and an occupant determination unit 53, when obtaining from the second face detection determination unit 52 a determination result indicating that the face of an occupant has been detected in the first range 6. an occupant determination function that, when a determination result indicating that the face of an occupant could not be detected is acquired, determines that the first face, which is the face of an occupant detected in the first captured image, is a face other than that of the driver if the first face is located in the second range 7 of the first captured image, and that the first face is the face of the driver if the first face is not located in the second range 7 of the first captured image; an abnormal attitude determination function that, when the abnormal attitude determination unit 54 acquires a determination result indicating that the first face is a face other than that of the driver from the occupant determination unit 53, determines whether or not the driver has an abnormal attitude using a third captured image among the multiple captured images that was captured before the first captured image and for which a determination result indicating that the face of an occupant could be detected was output by the first face detection determination unit 51; and an abnormal state determination function that the abnormal state determination unit 55 uses the determination result of the abnormal attitude determination unit 54 to determine whether or not the driver is in an abnormal state. When the occupant's face disappears from the first range 6 and then reappears in the first range 6, the occupant state determination program uses the second range 7 to determine whether the first face is a face other than that of the driver. With this configuration, even if the driver becomes abnormal and disappears from the first range 6, and another occupant attempts to assist the driver and reappears in the first range 6 from the same direction as the driver disappeared, it can be determined that the occupant's face that appeared in the first range 6 is a face other than that of the driver.Therefore, it is possible to avoid a situation in which information about the face of a person other than the driver, which is not the target of abnormal state determination, is used for abnormal posture determination, thereby improving the accuracy of detecting the occupant state in the occupant state determination program.

[0104] In this embodiment, an example has been shown in which the most recent captured image among the captured images acquired by the image acquisition unit 11 is set as the first captured image, but this is only one example, and the first captured image may be an image captured at any point in time among the captured images acquired by the image acquisition unit 11.

[0105] 1 Occupant state determination device, 2 Imaging device, 3 Output device, 11 Image acquisition unit, 12 Face detection unit, 13 Feature detection unit, 14 Reference feature estimation unit, 15 Determination unit, 31 Face direction detection unit, 32 Head position detection unit, 33 Part position detection unit, 34 Detection reliability calculation unit, 51 First face detection determination unit, 52 Second face detection determination unit, 53 Occupant determination unit, 54 Abnormal posture determination unit, 55 Abnormal state determination unit 100 Occupant state determination system, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory

Claims

1. A face detection unit that detects the face of the occupant in a plurality of captured images each including both a first range where the face of the driver among the occupants of the vehicle can exist and a second range where faces other than the driver can exist; A first face detection determination unit that makes a first face detection determination as to whether the face of the occupant can be detected in the first range in the plurality of captured images; When a determination result indicating that the face of the occupant has been detected in a first captured image among the plurality of captured images is obtained from the first face detection determination unit, a second face detection determination as to whether the face of the occupant can be detected in the first range is made using a second captured image captured before the first captured image among the plurality of captured images, by a second face detection determination unit; When a determination result indicating that the face of the occupant could not be detected is obtained from the second face detection determination unit, if a first face, which is the face of the occupant detected in the first captured image, is located in the second range of the first captured image, the first face is determined to be a face other than the driver, and if the first face is not located in the second range of the first captured image, an occupant determination is made that the first face is the face of the driver, by an occupant determination unit; When a determination result indicating that the first face is a face other than the driver is obtained from the occupant determination unit, an abnormal posture determination as to whether the driver is in an abnormal posture is made using a third captured image that was captured before the first captured image among the plurality of captured images and for which a determination result indicating that the face of the occupant has been detected by the first face detection determination unit has been output, by an abnormal posture determination unit; An abnormal state determination unit that makes an abnormal state determination as to whether the driver is in an abnormal state using the determination result of the abnormal posture determination unit; An occupant state determination device comprising the above.

2. When the abnormal posture determination unit obtains a determination result indicating that the face of the occupant has been detected from the second face detection determination unit, or when the abnormal posture determination unit obtains a determination result indicating that the first face is the face of the driver from the occupant determination unit, the abnormal posture determination unit makes the abnormal posture determination using the first captured image; When the abnormal posture determination unit obtains a determination result indicating that the face of the occupant could not be detected from the first face detection determination unit, the abnormal posture determination unit makes the abnormal posture determination using the third captured image. The occupant state determination device according to Claim 1.

3. Using the face of the occupant detected by the face detection unit, a feature amount of the face of the occupant is detected, and further, a first feature amount, which is a specific feature amount, is determined from the detected feature amount, and a second feature amount derived from the feature amount is calculated, and a feature amount detection unit; The feature amount includes a head position of the occupant, a face orientation of the occupant, and a part position of the face of the occupant; The first feature amount includes at least one of a head position of the occupant and a face orientation of the occupant; The second feature amount includes at least one of a face detection reliability indicating a detection reliability of at least one of a head position of the occupant and a face orientation of the occupant, and a face part detection reliability indicating a detection reliability of the part position; The face detection reliability is derived from at least one of a head position of the occupant and a face orientation of the occupant; The face part detection reliability is derived from the part position; The occupant state determination device according to claim 1 or 2.

4. The first face detection determination unit performs the first face detection determination in the first captured image using a first face detection feature amount, which is the second feature amount calculated by the feature amount detection unit in the first captured image; The second face detection determination unit performs the second face detection determination using a second face detection feature amount, which is the second feature amount calculated by the feature amount detection unit in the second captured image; The occupant state determination device according to claim 3.

5. The occupant determination unit performs the occupant determination using at least a first determination feature amount, which is the first feature amount detected by the feature amount detection unit in the first captured image; The occupant state determination device according to claim 3.

6. The occupant determination unit determines whether a first head position, which is the first determination feature amount, corresponds to the second range; When the first head position corresponds to the second range, the occupant determination unit determines that the first face is a face other than the driver; When the first head position does not correspond to the second range, the occupant determination unit determines that the first face is a face of the driver; The occupant state determination device according to claim 5.

7. The occupant determination unit determines whether a first head position, which is the first determination feature amount, corresponds to the second range, and whether an absolute value of a first face orientation, which is the first determination feature amount, is less than a first threshold value; When the first head position corresponds to the second range and the absolute value of the first face orientation is less than the first threshold value, the passenger determination unit determines that the first face is a face other than the driver's face. When the first head position does not correspond to the second range or the absolute value of the first face orientation is greater than or equal to the first threshold value, the passenger determination unit determines that the first face is the driver's face. The passenger state determination device according to claim 5.

8. The passenger determination unit performs the passenger determination using the first determination feature amount and a third determination feature amount which is the first feature amount detected by the feature amount detection unit in the third captured image. The passenger state determination device according to claim 5.

9. The passenger determination unit determines whether or not the first head position which is the first determination feature amount corresponds to the second range, and whether or not the absolute value of the head position difference which is the difference between the first head position and the second head position which is the third determination feature amount is greater than or equal to a second threshold value. When the first head position corresponds to the second range and the absolute value of the head position difference is greater than or equal to the second threshold value, the passenger determination unit determines that the first face is a face other than the driver's face. When the first head position does not correspond to the second range or the absolute value of the head position difference is less than the second threshold value, the passenger determination unit determines that the first face is the driver's face. The passenger state determination device according to claim 8.

10. The passenger determination unit determines whether or not the first head position which is the first determination feature amount corresponds to the second range, and whether or not the absolute value of the face orientation difference which is the difference between the first face orientation which is the first determination feature amount and the second face orientation which is the third determination feature amount is greater than or equal to a third threshold value. When the first head position corresponds to the second range and the absolute value of the face orientation difference is greater than or equal to the third threshold value, the passenger determination unit determines that the first face is a face other than the driver's face. When the first head position does not correspond to the second range or the absolute value of the face orientation difference is less than the third threshold value, the passenger determination unit determines that the first face is the driver's face. The passenger state determination device according to claim 8.

11. The occupant determination unit determines whether the first head position, which is the first determination feature quantity, corresponds to the second range, and whether first recognition information, which is information for recognizing the first face, matches driver recognition information, which is information for recognizing the face of the driver. When the first head position corresponds to the second range and the first recognition information does not match the driver recognition information, the occupant determination unit determines that the first face is a face other than the driver's face. When the first head position does not correspond to the second range or the first recognition information matches the driver recognition information, the occupant determination unit determines that the first face is the face of the driver. The occupant state determination device according to claim 5.

12. The first range includes a range where the face of the driver sitting in the driver's seat of the vehicle is located, and the second range includes a range where the face of another occupant, who is an occupant other than the driver sitting in a seat other than the driver's seat, is located. The occupant state determination device according to claim 1 or 2.

13. An occupant state determination system including the occupant state determination device according to claim 1 or 2, an imaging device installed in the vehicle so as to be able to image at least the face of the occupant, and an output device mounted on the vehicle and configured to output a warning for the occupant being in an abnormal state. The occupant state determination device uses the captured image captured by the imaging device to determine whether the occupant is in the abnormal state, and when it is determined that the occupant is in the abnormal state, causes the output device to output a warning for the occupant being in the abnormal state. Occupant state determination system.

14. A face detection step in which a face detection unit detects the face of the occupant in a plurality of captured images each including both a first range, which is a range where the face of the driver among the occupants of the vehicle may exist, and a second range, which is a range where the face of an occupant other than the driver may exist. A first face detection determination step in which a first face detection determination unit performs a first face detection determination as to whether the face of the occupant can be detected in the first range in the plurality of captured images. When the second face detection determination unit obtains, from the first face detection determination unit, a determination result indicating that the face of the occupant has been detected in the first captured image among the plurality of captured images, a second face detection determination step of performing a second face detection determination as to whether the face of the occupant can be detected within the first range using a second captured image captured before the first captured image among the plurality of captured images; When the occupant determination unit obtains, from the second face detection determination unit, a determination result indicating that the face of the occupant could not be detected, if a first face, which is the face of the occupant detected in the first captured image, is located within the second range of the first captured image, the first face is determined to be a face other than the driver, and if the first face is not located within the second range of the first captured image, an occupant determination step of determining that the first face is the face of the driver; When the abnormal posture determination unit obtains, from the occupant determination unit, a determination result indicating that the first face is a face other than the driver, an abnormal posture determination step of performing an abnormal posture determination as to whether the driver is in an abnormal posture using a third captured image captured before the first captured image among the plurality of captured images and for which a determination result indicating that the face of the occupant has been detected by the first face detection determination unit has been output; An abnormal state determination step in which the abnormal state determination unit performs an abnormal state determination as to whether the driver is in an abnormal state using the determination result of the abnormal posture determination unit; An occupant state determination method comprising the above. [

15. ] In a computer, a face detection unit having a face detection function of detecting the face of the occupant in a plurality of captured images each including both a first range, which is a range where the face of the driver among the occupants of the vehicle may exist, and a second range, which is a range where a face other than the driver may exist; a first face detection determination unit having a first face detection determination function of performing a first face detection determination as to whether the face of the occupant can be detected within the first range in the plurality of captured images; a second face detection determination unit having a second face detection determination function of performing a second face detection determination as to whether the face of the occupant can be detected within the first range using a second captured image captured before the first captured image among the plurality of captured images when the second face detection determination unit obtains, from the first face detection determination unit, a determination result indicating that the face of the occupant has been detected in the first captured image among the plurality of captured images; When the occupant determination unit obtains a determination result indicating that the face of the occupant cannot be detected from the second face detection determination unit, if the first face, which is the face of the occupant detected in the first captured image, is located in the second range of the first captured image, the first face is determined to be a face other than the driver. If the first face is not located in the second range of the first captured image, an occupant determination function is performed to determine that the first face is the face of the driver. When the abnormal posture determination unit obtains a determination result indicating that the first face is a face other than the driver from the occupant determination unit, using the third captured image that was captured before the first captured image among the plurality of captured images and for which a determination result indicating that the face of the occupant was detected by the first face detection determination unit was output, an abnormal posture determination function is performed to determine whether the driver has an abnormal posture. When the abnormal state determination unit uses the determination result of the abnormal posture determination unit to perform an abnormal state determination as to whether the driver is in an abnormal state, an abnormal state determination function is performed. An occupant state determination program for realizing the above.