Occupant state determination device, occupant state determination system, occupant state determination method, and occupant state determination program

Abstract

An occupant state determination device (1) according to the present disclosure comprises: a face detection unit (12) for detecting the face of an occupant in a captured image that includes a first range (6) being the range in which the face of a driver can exist and a second range (7) being the range in which the face of an occupant other than the driver can exist; a first face detection determination unit (51) for performing a first face detection determination in the captured image; a second face detection determination unit (52) for performing a second face detection determination by using a second captured image captured prior to the first captured image in which the face of the occupant was detected by the first face detection determination unit (51); an occupant determination unit (53) for performing an occupant determination that, in the case where a first face being the face of the occupant detected in the first captured image is located in the second range, the first face is the face of an occupant other than the driver; an abnormal posture determination unit (54) for performing an abnormal posture determination by using a third captured image which was captured prior to the first captured image and from which the face of the occupant was detected by the first face detection determination unit; and an abnormal state determination unit (55) for performing an abnormal state determination.

Description

[Technical Field]

[0001] The present disclosure relates to an occupant state determination device, an occupant state determination system, an occupant state determination method, and an occupant state determination program. [Background technology]

[0002] A technology has been developed that captures an image of the interior of a vehicle and uses information extracted from the occupant in the captured image to detect whether the occupant is in an abnormal state.

[0003] For example, in the abnormality determination device of Patent Document 1, when a person disappears from an operable range that is preset as a range in which the steering of a vehicle can be operated and then reappears in the operable range, if the appearance position where the person appeared in the operable range and the disappearance position where the person disappeared from the operable range are not the same, it is determined that the person who appeared and the person who disappeared are not the same person, and the driver is in an abnormal state.Furthermore, if the appearance position and the disappearance position are the same, it is determined that the person who appeared and the person who disappeared are the same person, and the driver is not in an abnormal state. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-009179 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, there are various possible ways in which a person may disappear or appear in the operable range. Therefore, there may be an abnormal state of the driver that cannot be detected by a determination using the appearance position and disappearance position. For example, if the driver becomes abnormal and disappears from the operable range, and an occupant other than the driver reappears in the operable range from the same position as the driver's disappearance position to assist the driver, the abnormality determination device of Patent Document 1 determines that the driver is not in an abnormal state. Therefore, the abnormality determination device of Patent Document 1, which determines the driver's abnormal state based on a comparison between the appearance position and the disappearance position as described above, has a problem in that the accuracy of detecting the occupant's state is reduced because the abnormal state of the driver is not determined sufficiently.

[0006] The present disclosure has been made to solve such problems, and has an object to provide an occupant state determination device that improves the accuracy of detecting the occupant state. [Means for solving the problem]

[0007] The occupant state determination device according to the present disclosure has a first range in which the face of the driver of the vehicle occupants is likely to be present, and a range in which the faces of persons other than the driver are likely to be present. and having a range that overlaps with the first range.a face detection unit that detects the face of an occupant in a plurality of captured images each including both a first range and a second range; a first face detection determination unit that performs a first face detection determination as to whether or not the face of an occupant has been detected in the first range in the plurality of captured images; a second face detection determination unit that, when a determination result indicating that the face of an occupant has been detected in a first captured image of the plurality of captured images is obtained from the first face detection determination unit, performs a second face detection determination as to whether or not the face of an occupant has been detected in the first range using a second captured image of the plurality of captured images that was captured before the first captured image; and, when a determination result indicating that the face of an occupant has not been detected is obtained from the second face detection determination unit, performs a second face detection determination as to whether or not the face of an occupant has been detected in the first range. The system is equipped with an occupant determination unit that determines that the first face is a face other than that of the driver when the first face is located within a second range of the first captured image, and that the first face is a face of the driver when the first face is not located within the second range of the first captured image; an abnormal attitude determination unit that, when a determination result indicating that the first face is a face other than that of the driver is obtained from the occupant determination unit, determines whether or not the driver is in an abnormal attitude using a third captured image among the multiple captured images that was captured before the first captured image and for which a determination result indicating that the occupant's face was detected was output by the first face detection determination unit; and an abnormal state determination unit that determines whether or not the driver is in an abnormal state using the determination result of the abnormal attitude determination unit. [Effects of the Invention]

[0008] According to the occupant state determination device according to the embodiment of the present disclosure, it is possible to improve the accuracy of detecting the occupant state. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of an occupant state determination device according to a first embodiment. [Figure 2] 2 is a diagram showing the interior of a vehicle, a first range, and a second range according to the first embodiment. FIG. [Figure 3] 10A and 10B are diagrams for explaining a plurality of types of poor posture. [Figure 4]4 is a flowchart showing an example of the operation of the occupant state determination device according to the first embodiment. [Figure 5] 1 is a diagram illustrating an example of a hardware configuration of an occupant state determination device according to a first embodiment. [Figure 6] 1 is a diagram illustrating an example of a hardware configuration of an occupant state determination device according to a first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1 The occupant state determination device 1 according to this embodiment determines whether the posture of a vehicle occupant is abnormal based on a captured image of the vehicle occupant. In the following embodiment, as an example, the vehicle occupant for whom the occupant state determination device 1 determines whether or not the vehicle is in an abnormal state is the driver of the vehicle.

[0011] FIG. 1 is a block diagram showing the configuration of an occupant state determination device 1 according to this embodiment. FIG. 2 is a diagram showing the interior of a vehicle, a first range (to be described later), and a second range (to be described later). In this embodiment, it is assumed that the occupant state determination device 1 is mounted on a vehicle. However, the occupant state determination device 1 is not limited to being permanently installed in a vehicle, and may be implemented in a portable device that can be carried into a vehicle, such as a portable navigation device. Furthermore, a part or all of the occupant state determination device 1 may be implemented on a server that can communicate with the vehicle.

[0012] The occupant state determination device 1 is connected to an imaging device 2 and an output device 3, and the occupant state determination device 1, the imaging device 2, and the output device 3 constitute an occupant state determination system 100.

[0013] The imaging device 2 is mounted on a vehicle and is installed so as to be able to capture an image of an imaging range including both a first range 6, which is a range in which the driver's face may be present, and a second range 7, which is a range in which faces other than the driver's may be present. Here, the first range 6 is a range in which the face of a driver seated in the driver's seat to drive the vehicle is located. If the front left seat among the seats installed inside the vehicle is the driver's seat, the first range 6 is the range indicated by the dotted line in FIG. 2 and is set to include a headrest 61 of the driver's seat and a portion of the backrest 62 of the driver's seat. Note that this is merely an example, and the first range 6 may be appropriately set by an administrator of the occupant state determination device 1 or may be stored in advance in the occupant state determination device 1. The second range 7 will be described in detail later.

[0014] The imaging device 2 is assumed to be installed, for example, near the center of the instrument panel in the vehicle width direction or on the center console. For example, the imaging device 2 may be a device shared with a so-called DMS (Driver Monitoring System) that is installed for the purpose of monitoring the interior of the vehicle. The imaging device 2 is a visible light camera or an infrared camera. The imaging device 2 captures images of the interior of the vehicle at intervals of, for example, 30 to 60 fps (frames per second) and outputs the captured images to the image acquisition unit 11 of the occupant state determination device 1. When the imaging device 2 captures images of the interior of the vehicle at intervals of 30 fps, it captures 30 frames per second.

[0015] The occupant state determination device 1 determines whether or not the driver is in an abnormal state using the captured image captured by the imaging device 2. For example, the occupant state determination device 1 determines whether or not the driver is in an abnormal state for each frame. Details of the occupant state determination device 1 will be described later. When the occupant condition determination device 1 determines that the driver is in an abnormal condition, it causes the output device 3 to output a warning that the driver is in an abnormal condition.

[0016] The output device 3 is mounted on the vehicle and outputs a warning to the driver that an abnormal state is occurring. The output device 3 is, for example, an audio output device such as a speaker. The output device 3 may be provided, for example, in an audio device provided in a vehicle. For example, when information for outputting a warning (hereinafter referred to as "warning information") is output from the occupant state determination device 1, the output device 3 outputs a warning sound or a voice message notifying that the driver is in an abnormal state. Furthermore, the output device 3 may be, for example, a display device such as a display. For example, when warning information is output from the occupant state determination device 1, the output device 3 displays a message notifying that the driver is in an abnormal state.

[0017] The output device 3 may be, for example, a turn signal, a hazard lamp, or a headlight that is provided on the exterior of the vehicle so as to be visible from other vehicles. For example, the occupant state determination device 1 may output warning information to the output device 3 to notify people outside the vehicle, such as occupants of other vehicles, that the driver of the vehicle is in an abnormal state. The output device 3 may be mounted on, for example, another vehicle. The occupant state determination device 1 can transmit warning information to the output device 3 mounted on the other vehicle, and can output a voice or display a message to inform the occupants of the other vehicle that the driver of the vehicle is in an abnormal state.

[0018] As shown in FIG. 1, the occupant state determination device 1 includes an image acquisition unit 11, a face detection unit 12, a feature amount detection unit 13, a reference feature amount estimation unit 14, and a determination unit 1515. The feature amount detection unit 13 includes a face direction detection unit 31, a head position detection unit 32, a facial feature position detection unit 33, and a detection reliability calculation unit . The determination unit 15 includes a first face detection determination unit 51, a second face detection determination unit 52, an occupant determination unit 53, an abnormal posture determination unit 54, and an abnormal state determination unit 55.

[0019] The image acquisition unit 11 acquires a captured image from the imaging device 2. The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12.

[0020] The face detection unit 12 detects the face of the driver in the first area 6 of the captured image acquired by the image acquisition unit 11. Specifically, the face detection unit 12 detects the driver's facial feature points, which indicate the driver's facial parts, by using a known image recognition technique such as edge detection on the captured image. Note that the facial features include the corners and corners of the eyes, nose, mouth, eyebrows, chin, etc. The driver's facial feature points are represented by, for example, coordinates on the captured image. The face detection unit 12 also detects the driver's face area. The driver's face area is, for example, the smallest rectangle that encloses the outline of the driver's face. The driver's face area is represented, for example, by the coordinates of the four corners of the smallest rectangle on the captured image. The face detection unit 12 outputs information about the detected facial feature points and facial area of ​​the driver (hereinafter referred to as "face information") to the feature amount detection unit 13. The face information is, for example, a captured image to which information that can identify the facial feature points and facial area of ​​the driver is added.

[0021] The feature amount detection unit 13 detects the feature amounts of the driver's face using the driver's face detected by the face detection unit 12. The feature amounts are the driver's facial direction, the driver's head position, and the driver's facial features. In this embodiment, the driver's facial direction and head position detected by the feature amount detection unit 13 are, for example, the facial direction and head position in real space. In this embodiment, the positions of the driver's facial features detected by the feature amount detection unit 13 are, for example, the positions of the facial features in a captured image. In this embodiment, as an example, the driver's facial features are the driver's eyes. In the following description, it is assumed that the driver's facial features are the driver's eyes.

[0022] The face direction detection unit 31 detects the face direction of the driver in real space based on the face information output from the face detection unit 12, specifically, based on the driver's face in the captured image detected by the face detection unit 12. The face direction detection unit 31 may detect the face direction of the driver using, for example, a known face direction detection technique for detecting the face direction from a captured image. The face direction of the driver is represented, for example, by at least one of a yaw angle, which is a rotation angle about an axis in the vertical direction of the vehicle, a pitch angle, which is a rotation angle about an axis in the horizontal direction of the vehicle, and a roll angle, which is a rotation angle about an axis in the longitudinal direction of the vehicle, which are predetermined references. The face direction detection unit 31 outputs information regarding the face direction of the driver (hereinafter referred to as "face direction information") to the reference feature estimation unit 14 and the determination unit 15. The face direction information includes at least one of information on the yaw angle, the pitch angle, and the roll angle.

[0023] The head position detection unit 32 detects the head position of the driver in real space based on the face information output from the face detection unit 12, more specifically, based on the face of the driver in the captured image detected by the face detection unit 12. In this embodiment, the head position of the driver in the captured image is indicated, for example, by the center of the driver's eyebrows. The head position detection unit 32 detects, for example, a point in real space corresponding to the center of the driver's eyebrows in the captured image as the driver's head position. Note that this is merely an example, and the position of the driver's head in the captured image may also be indicated, for example, by the center of the driver's facial area or the center of a line connecting the inner corners of the driver's eyes. In this case, the head position detection unit 32 detects, for example, a point in real space corresponding to the center of the driver's facial area in the captured image or the center of a line connecting the inner corners of the driver's eyes as the driver's head position in real space.

[0024] The head position detection unit 32 may detect the driver's head position using, for example, a known coordinate transformation technique that transforms points on a captured image into points in real space. The driver's head position is expressed, for example, by coordinates in real space. The head position detection unit 32 outputs information related to the detected driver's head position (hereinafter referred to as "head position information") to the reference feature estimation unit 14 and the determination unit 15. The head position information includes coordinate information of the driver's head position.

[0025] The feature position detection unit 33 detects the position of the driver's eyes in the captured image based on the face information output from the face detection unit 12, more specifically, based on the face of the driver in the captured image detected by the face detection unit 12. The face detection unit 12 detects the positions of the driver's right eye and left eye in the captured image. In this embodiment, the position of the driver's eyes is the center of a line connecting the inner corner and outer corner of the driver's eye in the captured image. The feature position detection unit 33 may detect the position of the driver's eyes using, for example, a known image recognition technique. The position of the driver's eyes is represented, for example, by coordinates on the captured image. The feature position detection unit 33 outputs information related to the detected position of the driver's eyes (hereinafter referred to as "feature position information") to the determination unit 15. The feature position information includes coordinate information of the position of the driver's right eye and coordinate information of the position of the driver's left eye.

[0026] The detection reliability calculation unit 34 calculates face detection reliability indicating the reliability that at least one of the detected driver's face direction and head position is at least one of a predetermined driver's face direction and head position. The detection reliability calculation unit 34 also calculates face part detection reliability indicating the reliability that the detected face parts are predetermined face parts. Then, the detection reliability calculation unit 34 outputs the calculated face detection reliability and facial part detection reliability to the determination unit 15. The face detection reliability and facial part detection reliability can be calculated, for example, using the methods described in the following documents. In the following document, the face detection reliability is calculated from the degree of match between at least one of a face direction and a head position learned in advance from a large number of face images and at least one of face direction and head position parts extracted from an image of the driver. Furthermore, the facial part detection reliability is calculated from the degree of match between face parts learned in advance from a large number of face images and face parts extracted from an image of the driver. “RapidObjectDetectionusingaBoostedCascadeofSimpleFeatures”,2001,PaulViola,etal.

[0027] In the following description, the features detected by the feature detection unit 13 refer to the driver's facial orientation, the driver's head position, and the driver's facial features. Of the features detected by the feature detection unit 13, at least one of the driver's facial orientation and head position is defined as a first feature, and at least one of the face detection reliability and facial feature detection reliability derived from the features is defined as a second feature.

[0028] The reference feature estimation unit 14 estimates the driver's reference feature as a reference feature using at least one of the facial direction information and the head position information. Specifically, the reference feature estimation unit 14 estimates at least one of the driver's reference facial direction and head position as a reference feature using at least one of the driver's facial direction and head position detected during a predetermined period (hereinafter referred to as the "reference setting period"). The reference setting period is, for example, a period from when the vehicle ignition is turned on until a predetermined time (e.g., 10 seconds) has elapsed. Note that this is merely an example, and the reference setting period may be, for example, a predetermined time (e.g., 3 seconds) when the vehicle speed is equal to or higher than a predetermined speed (e.g., 25 km / h) and the steering angle is within a predetermined range (e.g., ±20 degrees).

[0029] In addition, at least one of the driver's face direction and head position is used to estimate the reference feature, but the reference feature may also be estimated using at least one of the driver's face direction and head position and the face detection reliability. For example, the reference feature estimation unit 14 estimates an average value of at least one of the driver's face direction and head position during a reference setting period as at least one of the driver's face direction and head position in the reference feature of the driver. The reference feature estimation unit 14 outputs information on the estimated reference feature of the driver (hereinafter referred to as "reference feature information") to the determination unit 15. The reference feature information includes information on at least one of the face direction and head position in the driver's reference feature. The face direction in the driver's reference feature is expressed by at least one of the yaw angle, pitch angle, and roll angle. The face direction and head position in the driver's reference feature is expressed, for example, by coordinates in real space. The reference feature information also includes coordinate information of the driver's head position as information on the driver's head position.

[0030] In the following description, the most recent captured image among the captured images acquired by the image acquisition unit 11 will be referred to as the first captured image. The first captured image may also be referred to as the current frame.

[0031] The first face detection determination unit 51 uses the second feature amount calculated by the feature amount detection unit 13 in the captured image to perform a first face detection determination as to whether or not the face of the occupant has been detected within the first range. In the following description, the process of the occupant state determining device when the captured image used for the first face detection determination is the first captured image will be described. For example, when the first face detection judgment unit 51 performs a first face detection judgment in a first captured image, the first face detection judgment unit 51 performs a first face detection judgment as to whether or not the face of an occupant has been detected in the first range 6 using the first face detection feature, which is the second feature calculated in the first captured image by the feature detection unit 13.

[0032] First, a first face detection determination will be described when using the first face detection reliability, which is the first face detection feature. The first face detection determination unit 51 determines whether the first face detection reliability is equal to or greater than a fourth threshold. If the first face detection reliability is equal to or greater than the fourth threshold, the first face detection determination unit 51 determines that the occupant's face has been detected, and if the first face detection reliability is less than the fourth threshold, the first face detection determination unit 51 determines that the occupant's face has not been detected.

[0033] When the first face detection reliability is equal to or greater than the fourth threshold, the first face detection determination unit 51 outputs a first determination result, which is a determination result including information indicating that the occupant's face was detected and information indicating the first captured image, to the second face detection determination unit 52, the occupant determination unit 53, and the abnormal posture determination unit 54. When the first face detection reliability is less than the fourth threshold, the first face detection determination unit 51 outputs a determination result including information indicating that the occupant's face was not detected and information indicating the first captured image to the abnormal posture determination unit 54.

[0034] Next, a first face detection determination will be described when using the first facial part detection reliability, which is the first face detection feature. The first face detection determination unit 51 determines whether the first facial part detection reliability is equal to or greater than a fifth threshold. If the first facial part detection reliability is equal to or greater than the fifth threshold, the first face detection determination unit 51 determines that the occupant's face has been detected, and if the first facial part detection reliability is less than the fifth threshold, the first face detection determination unit 51 determines that the occupant's face has not been detected.

[0035] When the first facial part detection reliability is equal to or greater than the fifth threshold, the first face detection determination unit 51 outputs a determination result including information indicating that the occupant's face was detected and information indicating the first captured image to the second face detection determination unit 52. When the first facial part detection reliability is less than the fifth threshold, the first face detection determination unit 51 outputs a determination result including information indicating that the occupant's face was not detected and information indicating the first captured image to the abnormal posture determination unit 54.

[0036] Although the example in which the first face detection determination is made using either the first face detection reliability or the first facial part detection reliability has been shown, the present invention is not limited to this, and the first face detection determination may be made using the first face detection reliability and the first facial part detection reliability. In this case, the first face detection determination unit 51 determines that the occupant's face has been detected if the first face detection reliability is equal to or greater than a fourth threshold and the first facial part detection reliability is equal to or greater than a fifth threshold, and determines that the occupant's face has not been detected if the first face detection reliability is less than the fourth threshold or the first facial part detection reliability is less than the fifth threshold.

[0037] When the second face detection determination unit 52 receives a determination result from the first face detection determination unit 51 indicating that the occupant's face has been detected in the first captured image, the second face detection determination unit 52 performs a second face detection determination to determine whether the occupant's face has been detected in the first range 6, using a second captured image captured before the first captured image. Specifically, the second face detection determination unit 52 performs the second face detection determination using a second face detection feature, which is a second feature calculated in the second captured image by the feature detection unit 13. Specifically, the second captured image is a captured image that is one frame before the first captured image. The second captured image may also be referred to as the immediately preceding frame.

[0038] First, the second face detection determination when using the second face detection reliability, which is the second face detection feature, will be described. The second face detection determination unit 52 determines whether the second face detection reliability is equal to or greater than a fourth threshold. If the second face detection reliability is equal to or greater than the fourth threshold, the second face detection determination unit 52 determines that the occupant's face has been detected, and if the second face detection reliability is less than the fourth threshold, the second face detection determination unit 52 determines that the occupant's face has not been detected.

[0039] If the second face detection reliability is equal to or greater than the fourth threshold, the second face detection determination unit 52 outputs a determination result including information indicating that the occupant's face was detected and information indicating the second captured image to the abnormal posture determination unit 54. If the second face detection reliability is less than the fourth threshold, the second face detection determination unit 52 outputs a determination result including information indicating that the occupant's face was not detected and information indicating the second captured image to the occupant determination unit 53.

[0040] Next, a second face detection determination will be described when using the second facial part detection reliability, which is the second face detection feature. The second face detection determination unit 52 determines whether the second facial part detection reliability is equal to or greater than a fifth threshold. If the second facial part detection reliability is equal to or greater than the fifth threshold, the second face detection determination unit 52 determines that the occupant's face has been detected, and if the second facial part detection reliability is less than the fifth threshold, the second face detection determination unit 52 determines that the occupant's face has not been detected.

[0041] If the second facial part detection reliability is equal to or greater than the fifth threshold, the second face detection determination unit 52 outputs a determination result including information indicating that the occupant's face was detected and information indicating the second captured image to the abnormal posture determination unit 54. If the second facial part detection reliability is less than the fifth threshold, the second face detection determination unit 52 outputs a determination result including information indicating that the occupant's face was not detected and information indicating the second captured image to the occupant determination unit 53.

[0042] Although the example in which the second face detection determination is made using either the second face detection reliability or the second facial part detection reliability has been shown, the second face detection determination may be made using both the second face detection reliability and the second facial part detection reliability. In this case, the second face detection determination unit 52 determines that the occupant's face has been detected if the second face detection reliability is equal to or greater than a fourth threshold and the second facial part detection reliability is equal to or greater than a fifth threshold, and determines that the occupant's face has not been detected if the second face detection reliability is less than the fourth threshold or the second facial part detection reliability is less than the fifth threshold.

[0043] When the occupant determination unit 53 obtains a determination result from the second face detection determination unit 52 indicating that the occupant's face could not be detected, if the first face, which is the occupant's face detected in the first captured image, is located within a second range of the first captured image, it determines that the first face is a face other than that of the driver, and if the first face is not located within the second range of the first captured image, it determines that the first face is the face of the driver.

[0044] Here, the second range 7 will be described in the case where the face other than the driver is the face of another occupant other than the driver. The second range 7 is a range where the face of the other occupant is considered to be located. More specifically, the second range 7 includes a range where the faces of other occupants seated in seats other than the driver's seat of the vehicle are located, and a range where the faces of other occupants who attempt to assist the driver in an abnormal state are located. As shown in FIG. 2 , the second range 7 is a range indicated by a dashed line, and is set to include a portion of the backrest 62 of the driver's seat, headrests of seats other than the driver's seat, such as a passenger seat headrest 71 and a rear seat headrest 73, and portions of the backrests of seats other than the driver's seat, such as the passenger seat backrest 72 and the rear seat backrest 74. Note that this is merely an example, and the second range 7 may be appropriately set by an administrator of the occupant state determination device 1 or may be stored in advance in the occupant state determination device 1.

[0045] In this embodiment, an example has been given in which the faces other than the driver are the faces of other passengers, but this is not limited to this, and faces other than the driver also include patterns with shapes or structural features similar to those of a human face, specifically faces printed on masks or clothing, landscape patterns, etc.

[0046] Here, the reason why the occupant determination unit 53 performs occupant determination will be explained. First, when the occupant determination unit 53 obtains a determination result from the second face detection determination unit 52 indicating that the occupant's face could not be detected (hereinafter referred to as "Case A"), this refers to a case where the occupant's face can be detected in the first area 6 set in the first captured image but not in the first area 6 set in the second captured image. As described above, the second captured image is captured before the first captured image. In other words, Case A refers to a case where the occupant's face can be detected again after the occupant state determination device 1 can no longer detect the occupant's face (hereinafter referred to as "face lost return"). For example, the face lost return includes a state where the driver disappears from the first area 6 to pick up an object near the driver's seat and then reappears in the first area 6, a state where the driver becomes abnormal and disappears from the first area 6, and another occupant reappears in the first area 6 to assist the driver, etc. In other words, when face loss recovery occurs, there is a possibility that the face of an occupant detected in the first range 6 set in the first captured image is a face other than that of the driver. Therefore, if the occupant determination unit 53 does not perform occupant determination, an abnormal posture determination (described later) is performed using information about faces other than that of the driver. In other words, if information about faces other than that of the driver, which are not the subject of the occupant state determination device 1's determination of whether or not there is an abnormal state, is used in the abnormal posture determination, the accuracy of the abnormal state determination (described later) performed using the determination result of the abnormal posture determination decreases. As a result, there is a possibility that the accuracy of detecting the occupant state in the occupant state determination device 1 decreases. For the above reasons, the occupant determination unit 53 performs occupant determination.

[0047] Specifically, the occupant determination unit 53 determines whether or not the face of the occupant is located at least in the second range 7, thereby performing the occupant determination. In this embodiment, five types of occupant determination will be described, which are performed by the occupant determination unit 53. The occupant determination unit 53 performs one of the five occupant determinations.

[0048] In the following description, the first feature detected by the feature detection unit 13 in the first captured image will be referred to as the first judgment feature, and the first feature detected by the feature detection unit 13 in the third captured image, which is an image captured before the first captured image and for which the first face detection judgment unit 51 has output a judgment result indicating that the occupant's face has been detected, will be referred to as the third judgment feature.

[0049] Here, the third captured image will be described. The third captured image is determined using the determination result output by the first face detection determination unit 51 before the occupant determination using the first captured image. Specifically, the third captured image is an image captured one or more frames before the first captured image, and is an image in which it is determined that the first face detection determination unit 51 has detected the face of an occupant.

[0050] For example, a case will be described in which the first face detection determination unit 51 determines that the occupant's face was detected in the first captured image, determines that the occupant's face was not detected in the captured image one to three frames before the first captured image, and determines that the occupant's face was detected in the first captured image four frames before the first captured image (hereinafter referred to as "case B"). In case B, the first face detection determination unit 51 outputs a determination result corresponding to the captured image four frames before the first captured image to the occupant determination unit 53, and then outputs a determination result corresponding to the first captured image. Then, in case A, the occupant determination unit 53 performs occupant determination using the captured image four frames before the first captured image as the third captured image.

[0051] In other words, Case B refers to a case where the occupant's face was detected in the image captured four frames before the first captured image, but was not detected in the images captured one to three frames before the first captured image, and was then able to be detected again in the first captured image. Here, a case where the occupant's face was previously detected but is now no longer detectable in the occupant state determination device 1 is referred to as a "face lost" occurrence, a case where the occupant's face can no longer be detected is referred to as "face lost," and, as described above, a case where the occupant's face becomes detectable again after it has become undetectable is referred to as "face lost recovery." The first captured image is the image captured after face lost recovery, the second captured image is the image captured during face lost, and the third captured image is the image captured before face lost occurred. In the following description, the third captured image may also be referred to as a frame before face lost occurred.

[0052] First, the occupant determination performed using the first head position, which is the first determination feature, will be described. The occupant determination unit 53 determines whether the first head position falls within the second range 7 or not. If the first head position falls within the second range 7, the occupant determination unit 53 determines that the first face is the face of a person other than the driver, and if the first head position does not fall within the second range 7, the occupant determination unit 53 determines that the first face is the face of the driver.

[0053] When the first head position falls within the second range 7, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54. If the first head position does not fall within the second range 7, the occupant determination unit 53 outputs a determination result including information indicating that the first face is the driver's face to the abnormal posture determination unit 54.

[0054] Next, the occupant determination performed using the first head position and first face direction, which are the first determination feature amounts, will be described. The occupant determination unit 53 determines whether the first head position falls within the second range 7 and whether the absolute value of the first face direction is less than a first threshold value. The occupant determination unit determines that the first face is a face other than that of the driver if the first head position falls within the second range and the absolute value of the first facial direction is less than a first threshold, and determines that the first face is the face of the driver if the first head position does not fall within the second range or the absolute value of the first facial direction is equal to or greater than the first threshold. Here, the third range may be appropriately set by a manager of the occupant state determination device 1 or may be stored in the occupant state determination device 1 in advance.

[0055] When the first head position falls within the second range 7 and the first facial orientation falls within the third range, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54. If the first head position does not fall within the second range 7 or the first facial orientation does not fall within the third range, the occupant determination unit 53 outputs a determination result including information indicating that the first face is the driver's face to the abnormal posture determination unit 54.

[0056] Next, an occupant determination performed using the first head position, which is the first determination feature amount, and the second head position, which is the third determination feature amount, will be described. The occupant determination unit 53 determines whether the first head position falls within the second range 7 and whether the absolute value of the head position difference, which is the difference between the first head position and the third head position, is equal to or greater than a second threshold value. If the first head position falls within the second range 7 and the absolute value of the head position difference is equal to or greater than the second threshold, the occupant determination unit 53 determines that the first face is the face of a person other than the driver, and if the first head position does not fall within the second range 7 or the absolute value of the head position difference is less than the second threshold, the occupant determination unit 53 determines that the first face is the face of the driver. Here, the second threshold value may be appropriately set by a manager of the occupant state determination device 1 or may be stored in the occupant state determination device 1 in advance.

[0057] If the first head position falls within the second range 7 and the absolute value of the head position difference is greater than or equal to the second threshold, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54. If the first head position does not fall within the second range 7 or the absolute value of the head position difference is less than the second threshold, the occupant determination unit 53 outputs a determination result including information indicating that the first face is the driver's face to the abnormal posture determination unit 54.

[0058] Next, an occupant determination performed using the first head position and first face direction, which are the first determination feature amounts, and the second face direction, which is the third determination feature amount, will be described. The occupant determination unit 53 determines whether the first head position falls within the second range 7 and whether the absolute value of the facial direction difference, which is the difference between the first facial direction and the third facial direction, is equal to or greater than a third threshold value. The occupant determination unit 53 determines that the first face is a face other than that of the driver if the first head position falls within the second range 7 and the absolute value of the facial direction difference is equal to or greater than the third threshold, and determines that the first face is the face of the driver if the first head position does not fall within the second range 7 or the absolute value of the facial direction difference is less than the third threshold. Here, the third threshold value may be appropriately set by a manager of the occupant state determination device 1 or may be stored in the occupant state determination device 1 in advance.

[0059] If the first head position falls within the second range 7 and the absolute value of the facial direction difference is greater than or equal to the third threshold, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54. If the first head position does not fall within the second range 7 or the absolute value of the facial direction difference is less than the third threshold, the occupant determination unit 53 outputs a determination result including information indicating that the first face is the driver's face to the abnormal posture determination unit 54.

[0060] Next, we will explain occupant determination performed using the first head position, which is the first determination feature, and recognition information acquired from a recognition unit (not shown) configured to recognize a person in the first captured image. The occupant determination unit 53 determines whether or not the first recognition information, which is information for recognizing the first face, matches the driver recognition information, which is information for recognizing the face of the driver. If the first head position falls within the second range 7 and the first recognition information does not match the driver recognition information, the occupant determination unit 53 determines that the first face is the face of a person other than the driver, and if the first head position does not fall within the second range 7 or the first recognition information matches the driver recognition information, the occupant determination unit 53 determines that the first face is the face of the driver.

[0061] If the first head position falls within the second range 7 and the first recognition information does not match the driver identification information, the occupant determination unit 53 outputs a determination result including information indicating that the first face is a face other than that of the driver to the abnormal posture determination unit 54. If the first head position does not fall within the second range 7 or the first recognition information matches the driver identification information, the occupant determination unit 53 outputs a determination result including information indicating that the first face is the driver's face to the abnormal posture determination unit 54.

[0062] When the abnormal posture judgment unit 54 obtains a judgment result from the second face detection judgment unit 52 indicating that the face of an occupant has been detected, or when it obtains a judgment result from the occupant judgment unit 53 indicating that the first face is the face of the driver, it performs an abnormal posture judgment using the first captured image. When the abnormal posture determination unit 54 obtains a determination result from the first face detection determination unit 51 indicating that the occupant's face could not be detected, or when it obtains a determination result from the occupant determination unit 53 indicating that the first face is a face other than that of the driver, the abnormal posture determination unit 54 performs an abnormal posture determination using the third captured image.

[0063] Specifically, when the abnormal posture judgment unit 54 obtains a judgment result from the second face detection judgment unit 52 indicating that the occupant's face has been detected, or when it obtains a judgment result from the occupant judgment unit 53 indicating that the first face is the driver's face, the abnormal posture judgment unit 54 performs an abnormal posture judgment using the first judgment feature. When the abnormal posture judgment unit 54 obtains a judgment result from the first face detection judgment unit 51 indicating that the occupant's face could not be detected, or when it obtains a judgment result from the occupant judgment unit 53 indicating that the first face is a face other than that of the driver, the abnormal posture judgment unit 54 performs an abnormal posture judgment using the third judgment feature.

[0064] Here, we will explain why the abnormal posture judgment unit 54 performs abnormal posture judgment using the first judgment feature when it obtains a judgment result from the second face detection judgment unit 52 indicating that an occupant's face has been detected, or when it obtains a judgment result from the occupant judgment unit 53 indicating that the first face is the driver's face.

[0065] First, when the abnormal posture determination unit 54 obtains a determination result from the second face detection / determination unit 52 indicating that the occupant's face has been detected (hereinafter referred to as "Case C"), this refers to a case where the occupant's face is detected in the first range 6 set in the first captured image and the occupant's face is detected in the first range 6 set in the second captured image. In other words, Case C refers to a state where the occupant's face is continuously detected in the occupant state determination device 1. Therefore, the occupant's face detected in the first range 6 set in the first captured image is considered to be the driver's face. Therefore, the abnormal posture determination unit 54 performs abnormal posture determination using the first determination feature.

[0066] Next, when the abnormal posture determination unit 54 acquires a determination result from the occupant determination unit 53 indicating that the first face is the face of the driver, this refers to a case where the face of the occupant detected again is the face of the driver after face loss recovery. Therefore, the face of the occupant detected in the first range 6 set in the first captured image is considered to be the face of the driver. Therefore, the abnormal posture determination unit 54 performs abnormal posture determination using the first determination feature amount.

[0067] Next, we will explain why the abnormal posture judgment unit 54 performs abnormal posture judgment using the third judgment feature when it obtains a judgment result from the first face detection judgment unit 51 indicating that the occupant's face could not be detected, or when it obtains a judgment result from the occupant judgment unit 53 indicating that the first face is the face of a face other than that of the driver.

[0068] First, when the abnormal posture determination unit 54 acquires a determination result from the first face detection determination unit 51 indicating that the occupant's face could not be detected, this refers to a case where face loss has occurred or the face has been lost. In other words, since the occupant's face could not be detected in the first range 6 set in the first captured image, the abnormal posture determination unit 54 performs abnormal posture determination using information about the occupant's face detected in the first range 6 set in the third captured image. Therefore, the abnormal posture determination unit 54 performs abnormal posture determination using the third determination feature amount.

[0069] Next, when the abnormal posture determination unit 54 acquires a determination result from the occupant determination unit 53 indicating that the first face is a face other than that of the driver, this refers to a case where the face of the occupant detected again is a face other than that of the driver after face loss recovery. Therefore, the face of the occupant detected in the first range 6 set in the first captured image is considered to be a face other than that of the driver. Furthermore, since the face of the occupant cannot be detected in the first range 6 set in the second captured image, the abnormal posture determination unit 54 performs abnormal posture determination using information about the face of the occupant detected in the first range 6 set in the third captured image. Therefore, the abnormal posture determination unit 54 performs abnormal posture determination using the third determination feature amount.

[0070] The abnormal posture determination unit 54 will be described in detail below. The abnormal posture determination unit 54 determines whether to use the first captured image or the third captured image as the determination image, which is the captured image used for abnormal posture determination, based on the determination result obtained from the first face detection determination unit 51, the second face detection determination unit 52, or the occupant determination unit 53. Then, the abnormal state determination unit 55 sets the first feature amount detected in the captured image adopted as the determination-use captured image as the determination-use feature amount. Then, the abnormal posture determination unit 54 determines whether the driver's posture corresponds to any of a plurality of predetermined posture imbalance types (hereinafter referred to as "posture imbalance types") by comparing the determination feature with the reference feature obtained from the reference feature estimation unit 14. The abnormal state determination unit 55 determines that the occupant has an abnormal posture if the driver's posture corresponds to any of the posture error types, and determines that the occupant does not have an abnormal posture if the driver's posture does not correspond to any of the posture error types.

[0071] Figure 3 (partially modified from the "Basic Design Document for Driver Abnormality Automatic Detection System" by the Advanced Safety Vehicle Promotion Study Group, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, March 2018) is a diagram for explaining several types of poor posture. As shown in Figure 3, there are several types of poor posture. Figure 3 shows the following: "head down" in which the driver leans forward and continues to have their face close to the steering wheel; "head down" in which the driver continues to have their face facing down; "backward lean" in which the driver's upper body leans backward and continues to have their face facing up; "backward lean" in which the driver's upper body leans up and continues to have their face facing up; "head tilted to the side" in which the driver's face is tilted to the left or right; "sideways lean" in which the driver's upper body leans to the left or right and their face is tilted in the same direction; and "leaning to the side" in which the driver's upper body leans to the left or right. In the present disclosure, the multiple pre-set types of posture abnormality are "head down," "head down," "backward arch," "arched back," "head tilted to the side only," "side tilt," and "leaning to the side," as shown in Fig. 3. Information on the multiple types of posture abnormality is stored in a location that can be referenced by the abnormal posture determination unit 54.

[0072] Specifically, the abnormal posture determination unit 54 determines which of a plurality of posture abnormality types the driver belongs to depending on how much the determination feature has changed compared to the reference feature. Note that the conditions for determining which posture abnormality type the driver belongs to based on how much the determination feature has changed compared to the reference feature are set in advance and stored in the abnormal posture determination unit 54.

[0073] If the driver's posture corresponds to one of the posture error types, the abnormal posture determination unit 54 outputs a determination result including information indicating that the driver has an abnormal posture to the abnormal state determination unit 55, and if the driver's posture does not correspond to one of the posture error types, the abnormal posture determination unit 54 outputs a determination result including information indicating that the driver does not have an abnormal posture to the abnormal state determination unit 55.

[0074] The abnormal state determination unit 55 uses the determination result output by the abnormal posture determination unit 54 to determine whether the driver is in an abnormal state.

[0075] When the abnormal posture determination unit 54 determines that the driver has an abnormal posture, the abnormal state determination unit 55 determines the duration of the state in which the driver has an abnormal posture. For example, the abnormal state determination unit 55 associates the determination result acquired from the abnormal posture determination unit 54 with information on the date and time when the determination result was acquired and stores the result in a storage unit (not shown). The abnormal state determination unit 55 may determine the duration of the state in which the driver has an abnormal posture from the stored determination result. When the duration of the state in which the driver has an abnormal posture reaches a sixth threshold, the abnormal state determination unit 55 determines that the driver is in an abnormal state.

[0076] The abnormal state determination unit 55 outputs to the output device 3 the determination result as to whether or not the driver is in an abnormal state.

[0077] Fig. 4 is a flowchart showing an example of the operation of the occupant state determination device 1 according to this embodiment. The processing executed by the occupant state determination device 1 according to this embodiment will be described with reference to the flowchart shown in Fig. 4. The operation of the occupant state determination device 1 is started, for example, after the ignition of the vehicle is turned on. It is also preferable to perform the occupant state determination processing every time an image is acquired from the imaging device 2, that is, for each frame.

[0078] The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST1). The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12.

[0079] After step ST1, the face detection unit 12 detects the face of the driver in the captured image acquired by the image acquisition unit 11 in step ST1 (step ST2). The face detection unit 12 outputs the face information to the feature amount detection unit 13 .

[0080] After step ST2, the face direction detection unit 31 detects the face direction of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The face direction detection unit 31 outputs face direction information to the reference feature amount estimation unit 14 and the determination unit 15.

[0081] After step ST2, the head position detection unit 32 detects the head position of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The head position detection unit 32 outputs the head position information to the reference feature estimation unit 14 and the determination unit 15.

[0082] After step ST2, the part position detection unit 33 detects the part positions of the driver's face in real space based on the face information output from the face detection unit 12 in step ST2, specifically based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The part position detection unit 33 outputs the part position information to the determination unit 15 .

[0083] After step ST3, the detection reliability calculation unit 34 calculates face detection reliability and facial part detection reliability based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST4). The detection reliability calculation unit 34 outputs the face detection reliability and the facial part detection reliability to the determination unit 15.

[0084] After step ST4, the reference feature amount estimation unit 14 estimates a reference feature amount using at least one of the face direction information and the head position information (step ST5). The detection reliability calculation unit 34 outputs the reference feature amount to the determination unit 15.

[0085] After step ST5, the first face detection determination unit 51 uses the first face detection feature amount to perform a first face detection determination as to whether or not the face of an occupant has been detected in the first range 6 (step ST6). Step ST6 indicates whether the first face detection determination unit 51 has succeeded in detecting the face of the occupant in the current frame.

[0086] If the first face detection judgment unit 51 is able to detect the occupant's face (if "YES" in step ST6), the second face detection judgment unit 52 uses the second captured image to perform a second face detection judgment as to whether the occupant's face was detected in the first range 6 (step ST7). Step ST7 indicates whether the second face detection determination unit 52 has succeeded in detecting the face of the occupant in the immediately preceding frame.

[0087] If the second face detection judgment unit 52 is unable to detect the face of an occupant (if "NO" in step ST7), the occupant judgment unit 53 uses the second range 7 to make an occupant judgment as to whether the first face is a face other than that of the driver (step ST8). Step ST9 means that the occupant determination unit 53 determines whether the face of the occupant detected in the current frame is a face other than that of the driver.

[0088] If the first face is a face other than that of the driver ("YES" in step ST8), the occupant determination unit 53 and the abnormal posture determination unit 54 use the frame before face loss occurs as the captured image for determination (step ST9).

[0089] If the second face detection judgment unit 52 is able to detect the face of an occupant (if "YES" in step ST7), or if the first face is the face of the driver (if "NO" in step ST8), the abnormal posture judgment unit 54 adopts the current frame as the captured image for judgment (step ST10).

[0090] If the first face detection judgment unit 51 cannot detect the occupant's face (if "NO" in step ST6), the occupant judgment unit 53 and the abnormal posture judgment unit 54 use the frame before face loss occurred as the captured image for judgment (step ST11).

[0091] After step ST9, step ST10, or step ST11, the abnormal posture determination unit 54 performs abnormal posture determination using the determination feature amount and the reference feature amount (step ST12).

[0092] If the abnormal posture determination unit 54 determines that the driver is in an abnormal posture (if "YES" in step ST12), the abnormal state determination unit 55 determines whether the duration of the driver's abnormal posture is equal to or greater than a sixth threshold value (step ST13).

[0093] If the duration of the state in which the driver is in an abnormal posture is equal to or longer than the sixth threshold value ("YES" in step ST13), the abnormal state determination unit 55 determines that the driver is in an abnormal state (step ST14).

[0094] If the abnormal posture determination unit 54 determines that the driver is not in an abnormal posture (if "NO" in step ST12), or if the duration of the driver's abnormal posture is less than the sixth threshold (if "NO" in step ST13), the abnormal state determination unit 55 determines that the driver is not in an abnormal state (step ST15).

[0095] 5 and 6 are diagrams showing an example of the hardware configuration of the occupant state determination device 1 according to the first embodiment. In the first embodiment, the functions of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference feature estimation unit 14, and the determination unit 15 are realized by a processing circuit 1001. That is, the occupant state determination device 1 includes the processing circuit 1001 for performing control to determine whether a vehicle occupant is in an abnormal state based on a captured image. The processing circuit 1001 may be dedicated hardware as shown in FIG. 5, or may be a processor 1004 that executes a program stored in a memory 1005 as shown in FIG. 6.

[0096] When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0097] When the processing circuit is the processor 1004, the functions of the image acquisition unit 11, face detection unit 12, feature amount detection unit 13, reference feature amount estimation unit 14, and determination unit 15 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 1005. The processor 1004 reads and executes the program stored in the memory 1005, thereby executing the functions of the image acquisition unit 11, face detection unit 12, feature amount detection unit 13, reference feature amount estimation unit 14, and determination unit 15. That is, the occupant state determination device 1 includes the memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST15 of FIG. 4 described above. It can also be said that the programs stored in the memory 1005 cause the computer to execute the processing procedures or methods of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference feature estimation unit 14, and determination unit 15. Here, the memory 1005 corresponds to, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a minidisk, a DVD (Digital Versatile Disc), or the like.

[0098] It is also possible to realize some of the functions of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference feature estimation unit 14, determination unit 15, and output control unit 16 with dedicated hardware and some with software or firmware. For example, the function of the image acquisition unit 11 can be realized by a processing circuit 1001 as dedicated hardware, and the functions of the face detection unit 12, feature detection unit 13, reference feature estimation unit 14, and determination unit 15 can be realized by the processor 1004 reading and executing programs stored in a memory 1005.

[0099] The storage unit (not shown) is configured by, for example, a memory 1005. The occupant state determination device 1 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the imaging device 2 or the output device 3.

[0100] The occupant state determination device 1 of this embodiment includes a face detection unit 12 that detects the face of an occupant in a plurality of captured images that each include both a first range 6, which is a range in which the face of the driver of the vehicle may be present, and a second range 7, which is a range in which the face of an occupant other than the driver may be present; a first face detection determination unit 51 that makes a first face detection determination as to whether or not the occupant's face has been detected in the first range 6 in the plurality of captured images; a second face detection determination unit 52 that, when a determination result indicating that the occupant's face has been detected in a first captured image of the plurality of captured images is obtained from the first face detection determination unit 51, makes a second face detection determination as to whether or not the occupant's face has been detected in the first range 6 using a second captured image of the plurality of captured images that was captured before the first captured image; and a determination result indicating that the occupant's face has not been detected from the second face detection determination unit 52. an occupant determination unit that, when a determination result indicating that the first face is a face other than the driver's is obtained from the occupant determination unit, determines that the first face is a face other than the driver's when the first face, which is the face of an occupant detected in the first captured image, is located in the second range 7 of the first captured image, and determines that the first face is a face of the driver when the first face is not located in the second range 7 of the first captured image; an abnormal attitude determination unit that, when a determination result indicating that the first face is a face other than the driver is obtained from the occupant determination unit, determines whether or not the driver is in an abnormal attitude using a third captured image, which was captured before the first captured image among the plurality of captured images and for which a determination result indicating that the face of an occupant was detected by the first face detection determination unit is output; and an abnormal state determination unit that uses the determination result of the abnormal attitude determination unit to determine whether or not the driver is in an abnormal state. When an occupant's face disappears from the first range 6 and then reappears in the first range 6, the occupant state determination device 1 uses the second range 7 to determine whether the first face is a face other than that of the driver. With this configuration, even if the driver becomes abnormal and disappears from the first range 6, and another occupant attempts to assist the driver and reappears in the first range 6 from the same direction as the driver's disappearance, the occupant state determination device 1 can determine that the occupant's face that appears in the first range 6 is a face other than that of the driver. This makes it possible to avoid a situation in which information about a face other than that of the driver that is not the target of abnormal state determination is used for abnormal posture determination. As a result, the occupant state determination device 1 can improve its detection accuracy of the occupant state.

[0101] The occupant status determination system 100 of this embodiment is equipped with the occupant status determination device 1 of this embodiment, an imaging device 2 installed in the vehicle so as to be able to capture an image of at least the face of the occupant, and an output device 3 mounted on the vehicle and outputting a warning that the occupant is in an abnormal state, where the occupant status determination device 1 uses the image captured by the imaging device 2 to determine whether the occupant is in an abnormal state, and if it determines that the occupant is in an abnormal state, causes the output device 3 to output a warning that the occupant is in an abnormal state. When an occupant's face disappears from the first range 6 and then reappears in the first range 6, the occupant state determination system 100 uses the second range 7 to determine whether the first face is a face other than that of the driver. With this configuration, even if the driver becomes abnormal and disappears from the first range 6, and another occupant attempts to assist the driver and reappears in the first range 6 from the same direction as the driver's disappearance, the occupant state determination system 100 can determine that the occupant's face that appears in the first range 6 is a face other than that of the driver. This makes it possible to avoid a situation in which information about a face other than that of the driver that is not the target of abnormal state determination is used for abnormal posture determination. As a result, the occupant state determination system 100 can improve the accuracy of detecting the occupant state.

[0102] The occupant state determination method of this embodiment includes a face detection step in which the face detection unit 12 detects the face of an occupant in a plurality of captured images each including both a first range 6, which is a range in which the face of the driver of the vehicle occupants can be present, and a second range 7, which is a range in which faces of persons other than the driver can be present; a first face detection determination step in which the first face detection determination unit 51 makes a first face detection determination as to whether the face of the occupant has been detected in the first range 6 in the plurality of captured images; a second face detection determination step in which the second face detection determination unit 52, when obtaining from the first face detection determination unit 51 a determination result indicating that the face of the occupant has been detected in a first captured image of the plurality of captured images, makes a second face detection determination as to whether the face of the occupant has been detected in the first range 6 using a second captured image of the plurality of captured images that was captured before the first captured image; and a second face detection determination step in which the occupant determination unit 53 determines whether the face of the occupant has been detected from the second face detection determination unit 52. an occupant determination step in which, when a determination result indicating that the first face was not detected is obtained from the occupant determination unit 53, the abnormal posture determination unit 54 determines that the first face is a face other than that of the driver if the first face, which is the face of an occupant detected in the first captured image, is located in the second range 7 of the first captured image, and determines that the first face is the face of the driver if the first face is not located in the second range 7 of the first captured image; an abnormal posture determination step in which, when the abnormal posture determination unit 54 obtains a determination result indicating that the first face is a face other than that of the driver from the occupant determination unit 53, the abnormal posture determination unit 54 performs an abnormal posture determination as to whether or not the driver is in an abnormal posture using a third captured image among the multiple captured images that was captured before the first captured image and for which a determination result indicating that the face of an occupant was detected was output by the first face detection determination unit 51; and an abnormal state determination step in which the abnormal state determination unit 55 uses the determination result of the abnormal posture determination unit 54 to determine whether or not the driver is in an abnormal state. In the occupant state determination method, when the occupant's face disappears from the first range 6 and reappears in the first range 6, the second range 7 is used to determine whether the first face is a face other than that of the driver. With this configuration, even if the driver becomes abnormal and disappears from the first range 6, and another occupant attempts to assist the driver and reappears in the first range 6 from the same direction as the driver's disappearance, it is possible to determine that the occupant's face that appeared in the first range 6 is a face other than that of the driver. This makes it possible to avoid a situation in which information about a face other than that of the driver that is not the target of abnormal state determination is used in abnormal posture determination. As a result, the occupant state detection accuracy can be improved in the occupant state determination method.

[0103] The occupant state determination program of the present embodiment includes a computer having a face detection function in which a face detection unit detects the face of an occupant in a plurality of captured images each including both a first range 6, which is a range in which the face of the driver of the vehicle occupants may be present, and a second range 7, which is a range in which the faces of persons other than the driver may be present; a first face detection determination function in which a first face detection determination unit 51 makes a first face detection determination as to whether or not the face of an occupant has been detected in the first range 6 in the plurality of captured images; a second face detection determination function in which a second face detection determination unit 52, when obtaining from the first face detection determination unit 51 a determination result indicating that the face of an occupant has been detected in a first captured image of the plurality of captured images, makes a second face detection determination as to whether or not the face of an occupant has been detected in the first range 6 using a second captured image of the plurality of captured images that was captured before the first captured image; and an occupant determination unit 53, when obtaining from the second face detection determination unit 52 a determination result indicating that the face of an occupant has been detected in a first captured image of the plurality of captured images, makes a second face detection determination as to whether or not the face of an occupant has been detected in the first range 6. an occupant determination function that, when a determination result indicating that the face of an occupant could not be detected is acquired from the occupant determination unit 53, determines that the first face is a face other than that of the driver if the first face, which is the face of an occupant detected in the first captured image, is located in the second range 7 of the first captured image, and that the first face is the face of the driver if the first face is not located in the second range 7 of the first captured image; an abnormal posture determination function that, when the abnormal posture determination unit 54 acquires a determination result indicating that the first face is a face other than that of the driver from the occupant determination unit 53, determines whether or not the driver is in an abnormal posture using a third captured image, which was captured before the first captured image among the multiple captured images and for which a determination result indicating that the face of an occupant could be detected was output by the first face detection determination unit 51; and an abnormal state determination function that, when the abnormal state determination unit 55 acquires a determination result indicating that the face of an occupant could not be detected, determines whether or not the driver is in an abnormal state using the determination result of the abnormal posture determination unit 54. When the occupant's face disappears from the first range 6 and then reappears in the first range 6, the occupant state determination program uses the second range 7 to determine whether the first face is a face other than that of the driver. With this configuration, even if the driver becomes abnormal and disappears from the first range 6, and another occupant attempts to assist the driver and reappears in the first range 6 from the same direction as the driver's disappearance, it can be determined that the occupant's face that appeared in the first range 6 is a face other than that of the driver. This makes it possible to avoid a situation in which information about faces other than the driver that are not subject to abnormal state determination is used in abnormal posture determination. As a result, the occupant state determination program can improve its detection accuracy for the occupant state.

[0104] In this embodiment, an example has been shown in which the most recent captured image among the captured images acquired by the image acquisition unit 11 is set as the first captured image, but this is just one example, and the first captured image may be an image captured at any point in time among the captured images acquired by the image acquisition unit 11. [Explanation of symbols]

[0105] 1 Occupant state determination device, 2 Imaging device, 3 Output device, 11 Image acquisition unit, 12 Face detection unit, 13 Feature detection unit, 14 Reference feature estimation unit, 15 Determination unit, 31 Face direction detection unit, 32 Head position detection unit, 33 Part position detection unit, 34 Detection reliability calculation unit, 51 First face detection determination unit, 52 Second face detection determination unit, 53 Occupant determination unit, 54 Abnormal posture determination unit, 55 Abnormal state determination unit 100 Occupant state determination system, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory

Claims

1. a face detection unit that detects the faces of the vehicle occupants in a plurality of captured images that each include both a first range, which is a range in which the face of a driver of the vehicle occupants may be present, and a second range, which is a range in which the faces of persons other than the driver may be present and has a range that overlaps with the first range; a first face detection determination unit that performs a first face detection determination as to whether or not the face of the occupant has been detected within the first range in the plurality of captured images; a second face detection determination unit that, when a determination result indicating that the face of the occupant has been detected in a first captured image of the plurality of captured images is obtained from the first face detection determination unit, performs a second face detection determination as to whether the face of the occupant has been detected in the first range using a second captured image of the plurality of captured images that has been captured before the first captured image; and an occupant determination unit that, when a determination result indicating that the face of the occupant could not be detected is obtained from the second face detection determination unit, determines that a first face, which is the face of the occupant detected in the first captured image, is a face other than that of the driver if the first face is located within the second range of the first captured image, and determines that the first face is the face of the driver if the first face is not located within the second range of the first captured image; an abnormal posture determination unit that, when a determination result indicating that the first face is a face other than that of the driver is obtained from the occupant determination unit, performs an abnormal posture determination as to whether the driver is in an abnormal posture using a third captured image among the plurality of captured images that was captured before the first captured image and for which a determination result indicating that the face of the occupant was detected was output by the first face detection determination unit; an abnormal state determination unit that determines whether the driver is in an abnormal state using a determination result of the abnormal posture determination unit; An occupant state determination device comprising:

2. the abnormal posture determination unit, when acquiring a determination result from the second face detection determination unit indicating that the face of the occupant has been detected, or when acquiring a determination result from the occupant determination unit indicating that the first face is the face of the driver, performs the abnormal posture determination using the first captured image; the abnormal attitude determination unit, when acquiring a determination result from the first face detection determination unit indicating that the face of the occupant could not be detected, performs the abnormal attitude determination using the third captured image. The occupant condition determination device according to claim 1 .

3. a feature amount detection unit that detects a feature amount of the occupant's face using the face of the occupant detected by the face detection unit, determines a first feature amount that is a specific feature amount from the detected feature amount, and calculates a second feature amount derived from the feature amount, the feature amount includes a head position of the occupant, a facial direction of the occupant, and a position of a facial feature of the occupant; the first feature amount includes at least one of a head position of the occupant and a facial orientation of the occupant, the second feature amount includes at least one of a face detection reliability indicating a detection reliability of at least one of a head position of the occupant and a facial orientation of the occupant, and a face part detection reliability indicating a detection reliability of the feature position, the face detection reliability is derived from at least one of a head position of the occupant and a facial orientation of the occupant; the facial feature detection reliability is derived from the facial feature positions; the first face detection determination unit performs the first face detection determination in the first captured image by using a first face detection feature amount, which is the second feature amount calculated in the first captured image by the feature amount detection unit; the second face detection determination unit performs the second face detection determination using a second face detection feature amount, which is the second feature amount calculated for the second captured image by the feature amount detection unit; and the occupant determination unit performs the occupant determination using at least a first determination feature amount, which is the first feature amount detected in the first captured image by the feature amount detection unit. The occupant condition determination device according to claim 1 or 2.

4. the occupant determination unit determines whether a first head position, which is the first determination feature, falls within the second range; the occupant determination unit determines that the first face is a face other than that of the driver when the first head position falls within the second range; the occupant determination unit determines that the first face is the face of the driver when the first head position does not fall within the second range. The occupant condition determination device according to claim 3.

5. the occupant determination unit determines whether a first head position, which is the first feature for determination, falls within the second range, and whether an absolute value of a first face direction, which is the first feature for determination and which becomes 0 when the face is turned in a direction facing the traveling direction of the vehicle or when the face is turned in a direction facing the center of an optical axis of an image capturing device installed in the vehicle so as to be able to capture an image of at least the face of the occupant, is less than a first threshold value; the occupant determination unit determines that the first face is a face other than that of the driver when the first head position falls within the second range and the absolute value of the first face direction is less than the first threshold value; the occupant determination unit determines that the first face is the face of the driver when the first head position does not fall within the second range or when the absolute value of the first face direction is equal to or greater than the first threshold value. The occupant condition determination device according to claim 3.

6. the occupant determination unit performs the occupant determination using the first determination feature amount and a third determination feature amount that is the first feature amount detected in the third captured image by the feature amount detection unit. The occupant condition determination device according to claim 3.

7. the occupant determination unit determines whether a first head position, which is the first determination feature, falls within the second range, and whether an absolute value of a head position difference, which is a difference between the first head position and a second head position, which is the third determination feature, is equal to or greater than a second threshold; the occupant determination unit determines that the first face is a face other than that of the driver when the first head position falls within the second range and the absolute value of the head position difference is equal to or greater than the second threshold; the occupant determination unit determines that the first face is the face of the driver when the first head position does not fall within the second range or when the absolute value of the head position difference is less than the second threshold value. The occupant state determination device according to claim 6.

8. the occupant determination unit determines whether a first head position, which is the first determination feature, falls within the second range, and whether an absolute value of a face direction difference, which is a difference between a first face direction, which is the first determination feature, and a second face direction, which is the third determination feature, is equal to or greater than a third threshold; the occupant determination unit determines that the first face is a face other than that of the driver when the first head position falls within the second range and the absolute value of the facial direction difference is equal to or greater than the third threshold; the occupant determination unit determines that the first face is the face of the driver when the first head position does not fall within the second range or when the absolute value of the facial direction difference is less than the third threshold value. The occupant state determination device according to claim 6.

9. the occupant determination unit determines whether a first head position, which is the first determination feature, falls within the second range, and whether first recognition information, which is information for recognizing the first face, matches driver recognition information, which is information for recognizing the face of the driver; the occupant determination unit determines that the first face is a face other than that of the driver when the first head position falls within the second range and the first recognition information does not match the driver recognition information, the occupant determination unit determines that the first face is the face of the driver when the first head position does not fall within the second range or when the first recognition information matches the driver recognition information; The occupant condition determination device according to claim 3.

10. the first range includes a range in which a face of the driver seated in the driver's seat of the vehicle is located, and the second range includes a range in which a face of another occupant other than the driver seated in a seat other than the driver's seat is located. The occupant condition determination device according to claim 1 or 2.

11. 3. An occupant state determination system comprising: the occupant state determination device according to claim 1 or 2; an imaging device installed in the vehicle so as to be able to capture an image of at least the face of the occupant; and an output device mounted in the vehicle and configured to output a warning that the occupant is in an abnormal state, the occupant state determination device determines whether the occupant is in the abnormal state using the captured image captured by the imaging device, and when it determines that the occupant is in the abnormal state, causes the output device to output a warning that the occupant is in the abnormal state. Occupant status determination system.

12. a face detection step in which a face detection unit detects the faces of the vehicle occupants in a plurality of captured images each including both a first range, which is a range in which the face of the driver of the vehicle occupants may be present, and a second range, which is a range in which the face of a person other than the driver may be present and has a range overlapping with the first range; a first face detection determination step in which a first face detection determination unit performs a first face detection determination as to whether or not the face of the occupant has been detected within the first range in the plurality of captured images; a second face detection determination step of performing a second face detection determination as to whether or not the face of the occupant has been detected within the first range, using a second captured image of the plurality of captured images that has been captured before the first captured image, when the second face detection determination unit acquires from the first face detection determination unit a determination result indicating that the face of the occupant has been detected in a first captured image of the plurality of captured images; an occupant determination step in which, when the occupant determination unit obtains a determination result from the second face detection determination unit indicating that the face of the occupant could not be detected, the occupant determination unit determines that a first face, which is the face of the occupant detected in the first captured image, is a face other than that of the driver if the first face is located within the second range of the first captured image, and determines that the first face is the face of the driver if the first face is not located within the second range of the first captured image; an abnormal posture determination step of, when the abnormal posture determination unit acquires from the occupant determination unit a determination result indicating that the first face is a face other than that of the driver, determining whether or not the driver is in an abnormal posture by using a third captured image among the plurality of captured images that was captured before the first captured image and for which a determination result indicating that the face of the occupant was detected was output by the first face detection determination unit; an abnormal state determination step in which an abnormal state determination unit determines whether the driver is in an abnormal state by using a determination result of the abnormal posture determination unit; An occupant state determination method comprising:

13. On the computer, a face detection function in which a face detection unit detects the faces of the vehicle occupants in a plurality of captured images each including both a first range in which the face of the driver of the vehicle may be present and a second range in which the face of a person other than the driver may be present, the second range overlapping with the first range; a first face detection determination function in which a first face detection determination unit performs a first face detection determination as to whether or not the face of the occupant has been detected within the first range in the plurality of captured images; a second face detection determination function that, when a second face detection determination unit acquires from the first face detection determination unit a determination result indicating that the face of the occupant has been detected in a first captured image of the plurality of captured images, performs a second face detection determination as to whether the face of the occupant has been detected in the first range using a second captured image of the plurality of captured images that has been captured before the first captured image; an occupant determination function that, when the occupant determination unit obtains a determination result from the second face detection determination unit indicating that the face of the occupant could not be detected, determines that a first face, which is the face of the occupant detected in the first captured image, is a face other than that of the driver if the first face is located within the second range of the first captured image, and determines that the first face is the face of the driver if the first face is not located within the second range of the first captured image; an abnormal posture determination function that, when the abnormal posture determination unit obtains a determination result from the occupant determination unit indicating that the first face is a face other than that of the driver, performs an abnormal posture determination as to whether the driver is in an abnormal posture by using a third captured image among the plurality of captured images that was captured before the first captured image and for which a determination result indicating that the face of the occupant was detected was output by the first face detection determination unit; an abnormal state determination function in which an abnormal state determination unit determines whether the driver is in an abnormal state by using the determination result of the abnormal posture determination unit; An occupant status determination program to achieve this.

Images