Abnormality detection device, abnormality detection system, abnormality detection method, and program

Abstract

Provided is an abnormality detection device that reduces the possibility of failure to detect an abnormality or erroneous detection of an abnormality more than before.　An abnormality detection device (1) is provided with: an image acquisition unit (3) that acquires a captured image of an occupant; a face orientation angle calculation unit (5) that calculates the occupant's face orientation angle from the captured image; a line-of-sight angle calculation unit (7) that calculates the occupant's line-of-sight angle from the captured image; a line-of-sight range determination unit (9) that determines whether or not the line-of-sight angle is within a predetermined line-of-sight range; a determination angle calculation unit (14) that, if the line-of-sight angle is within the predetermined line-of-sight range, uses the calculated face orientation angle to calculate a determination face orientation angle used to determine whether or not the state of the occupant is abnormal; and an abnormality determination unit (15) that determines that the state of the occupant is abnormal if the determination face orientation angle is not within a predetermined normal determination range.

Description

Anomaly detection device, anomaly detection system, anomaly detection method, and program

[0001] The present disclosure relates to an abnormality detection device, an abnormality detection system, an abnormality detection method, and a program for detecting an abnormality in an occupant.

[0002] In the prior art, a driver monitor system is disclosed that sets a zero point for the driver's facial direction angle based on the driver's facial direction angle recognized by a facial direction recognition unit when a zero point setting condition is met, and diagnoses the driver's condition based on the current facial direction angle relative to the set zero point (for example, Patent Document 1).

[0003] JP 2019-87150 A

[0004] However, the face direction angle and the gaze angle may be different angles, and even if the face direction angle is normal, the gaze angle may not be normal. When the gaze angle is not normal, it is desirable to determine that the occupant's condition is abnormal even if the face direction angle is normal. Also, even if the face direction angle is not normal, the gaze angle may be normal. When the gaze angle is normal, it is desirable to determine that the occupant's condition is not abnormal even if the face direction angle is not normal. In conventional technology, the gaze is not taken into consideration when setting the zero point of the face direction angle, which has the problem of the possibility of an abnormality not being detected or an abnormality being erroneously detected.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an anomaly detection device, an anomaly detection system, an anomaly detection method, and a program that reduce the possibility of an anomaly not being detected or an anomaly being falsely detected compared to conventional methods.

[0006] The abnormality detection device according to the present disclosure includes an image acquisition unit that acquires an image of an occupant, a face direction angle calculation unit that calculates the face direction angle of the occupant from the image, a gaze angle calculation unit that calculates the gaze angle of the occupant from the image, a gaze range determination unit that determines whether the gaze angle is within a predetermined gaze range, a determination angle calculation unit that calculates a determination face direction angle used to determine whether the state of the occupant is abnormal using the calculated face direction angle when the gaze angle is within the predetermined gaze range, and an abnormality determination unit that determines that the state of the occupant is abnormal when the determination face direction angle is not within a predetermined normal determination range.

[0007] An abnormality detection system according to the present disclosure includes the abnormality detection device and an imaging device that captures an image of an occupant and outputs the captured image to an image acquisition unit.

[0008] Furthermore, the abnormality detection method according to the present disclosure includes the steps of: an image acquisition unit acquiring an image of an occupant; a facial direction angle calculation unit calculating the facial direction angle of the occupant from the captured image; a gaze angle calculation unit calculating the gaze angle of the occupant from the captured image; a gaze range determination unit determining whether the gaze angle is within a predetermined gaze determination range; a determination angle calculation unit calculating a determination facial direction angle to be used for determining whether the occupant's condition is abnormal using the calculated facial direction angle when the gaze angle is within the predetermined gaze determination range; and an abnormality determination unit determining that the occupant's condition is abnormal when the determination facial direction angle is not within the predetermined normal determination range.

[0009] In addition, the program according to the present disclosure causes a computer to execute the following steps: an image acquisition unit acquiring an image of an occupant; a facial direction angle calculation unit calculating the facial direction angle of the occupant from the captured image; a gaze angle calculation unit calculating the gaze angle of the occupant from the captured image; a gaze range determination unit determining whether the gaze angle is within a predetermined gaze determination range; a determination angle calculation unit calculating a determination facial direction angle to be used for determining whether the occupant's condition is abnormal using the calculated facial direction angle when the gaze angle is within the predetermined gaze determination range; and an abnormality determination unit determining that the occupant's condition is abnormal if the determination facial direction angle is not within the predetermined normal determination range.

[0010] According to the anomaly detection device of the present disclosure, it is possible to reduce the possibility of an anomaly not being detected or an erroneous anomaly being detected. Furthermore, according to the anomaly detection system of the present disclosure, it is possible to reduce the possibility of an anomaly not being detected or an erroneous anomaly being detected. Furthermore, according to the anomaly detection method of the present disclosure, it is possible to reduce the possibility of an anomaly not being detected or an erroneous anomaly being detected. Furthermore, according to the program of the present disclosure, it is possible to reduce the possibility of an anomaly not being detected or an erroneous anomaly being detected.

[0011] FIG. 1 is a schematic diagram partially illustrating the interior of a vehicle equipped with an anomaly detection device according to embodiment 1 of the present disclosure. FIG. 2 is a schematic diagram partially illustrating the interior of a vehicle equipped with an anomaly detection device according to embodiment 1 of the present disclosure. FIG. 3 is a block diagram illustrating the configuration of an anomaly detection device and an anomaly detection system according to embodiment 1 of the present disclosure. FIG. 4 is a diagram illustrating an example hardware configuration of the anomaly detection device according to embodiment 1 of the present disclosure. FIG. 5 is a flowchart illustrating an example operation of the anomaly detection device according to embodiment 1 of the present disclosure. FIG. 6 is a flowchart illustrating an example operation of the anomaly detection device according to embodiment 2 of the present disclosure.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the drawings are schematic, and the relative sizes and positions shown in different drawings are not necessarily limited to those described and may be changed as appropriate. In the following description, similar components are denoted by the same reference numerals, and their names and functions are assumed to be the same or similar. Therefore, detailed descriptions thereof may be omitted.

[0013] In the Cartesian coordinate system XYZ shown in the figure, the vehicle's running direction is the X-axis direction, the vehicle width direction is the Y-axis direction, and the direction perpendicular to the X-axis and Y-axis is the Z-axis direction. The angle around the X-axis is the roll angle, the angle around the Y-axis is the pitch angle, and the angle around the Z-axis is the yaw angle. The above coordinate system will be referenced as appropriate in the following explanation.

[0014] First Embodiment An abnormality detection device 1 according to a first embodiment will be described with reference to Fig. 1 to Fig. 5. Fig. 1 is a schematic diagram partially showing the interior of a vehicle 100 equipped with an abnormality detection device 1 according to the first embodiment.

[0015] The abnormality detection device 1 is mounted on a vehicle 100. The vehicle 100 includes an imaging device 2, a sun visor 101, a front window 102, and a seat 103. In Fig. 1, an occupant 104 is riding in the vehicle 100.

[0016] The seat 103 is movable. That is, the seat position of the seat 103 can be changed depending on the physique of the occupant 104. The seat position of the seat 103 closest to the windshield 102 is called the FrontMost. The seat position of the seat 103 farthest from the windshield 102 is called the RearMost.

[0017] The imaging device 2 captures an image of the occupant. The imaging device 2 can capture an image of the occupant's face. The imaging device 2 is installed, for example, on an instrument panel, a steering column, or a rearview mirror. The imaging device 2 is installed, for example, near a location where a car navigation system is installed. The imaging device 2 does not move with the movable seat 103, for example. The imaging device 2 can be, for example, an infrared camera, a visible light camera, or the like.

[0018] Fig. 1 shows the interior of a vehicle 100 when the seat position is frontmost. Arrow A in Fig. 1 indicates the face direction angle. Fig. 1 illustrates a case where the pitch angle of the face direction angle is 50°. The face direction angle refers to the angle at which the face is facing, and can be expressed using, for example, a roll angle, a pitch angle, and a yaw angle.

[0019] As shown in Fig. 1, when the seat position is frontmost and the pitch angle of the face direction angle is 50°, the face is facing the direction of the windshield 102. Since the face is facing the windshield 102, it can be inferred that the line of sight is also facing the direction of the windshield 102, and therefore it can be assumed that the driver, who is the occupant, is not looking away while driving. In other words, it can be determined that the face direction angle of the occupant is normal. In other words, it can be determined that the occupant's condition is not abnormal.

[0020] Fig. 2 is a schematic diagram partially illustrating the interior of a vehicle 100 equipped with the anomaly detection device 1 according to embodiment 1. Fig. 2 illustrates the interior of the vehicle 100 when the seat position is rearmost. Arrow B shown in Fig. 2 indicates the face direction angle. Like Fig. 1, Fig. 2 illustrates an example in which the pitch angle of the face direction angle is 50°.

[0021] As shown in Figure 2, when the seat position is rearmost and the pitch angle of the face direction angle is 50°, the face is facing the direction of the sun visor 101. If the face is facing the direction of the sun visor 101, it can be inferred that the line of sight is also facing the direction of the sun visor 101, so it can be assumed that the driver, who is the occupant, is looking away while driving. In other words, it can be determined that the face direction angle of the occupant is not a normal face direction angle. In other words, it can be determined that the state of the occupant is abnormal.

[0022] The face direction angle shown in FIG. 1 and the face direction angle shown in FIG. 2 are the same angle, but whether they are normal or not differ. That is, the normal face direction angle relative to the image capture device 2 differs depending on the seat position, the occupant's physique and sitting posture, etc. Because the normal face direction angle relative to the image capture device 2 differs, there is a possibility that an abnormality will not be detected, that is, an abnormality will not be detected when it should be determined. There is also a possibility that an abnormality will be falsely detected, that is, an abnormality will be detected when it should not be determined. Therefore, the abnormality detection device 1 according to the first embodiment corrects the face direction angle used when determining whether the occupant's condition is abnormal, even when the normal face direction angle differs due to differences in the seat position, the occupant's physique and sitting posture, etc.

[0023] The term "occupant" includes the driver. An occupant's abnormal condition includes, for example, the driver being distracted while driving. An occupant's abnormal condition also includes the occupant being physically unwell.

[0024] 3 is a block diagram showing the configuration of an anomaly detection device 1 according to embodiment 1. The anomaly detection device 1 includes an image acquisition unit 3, a face information detection unit 4, a face direction angle calculation unit 5, a gaze information detection unit 6, a gaze angle calculation unit 7, an effective gaze determination unit 8, a gaze range determination unit 9, a face direction range determination unit 10, a storage unit 11, a frequency distribution creation unit 12, a front face direction angle calculation unit 13, a determination angle calculation unit 14, an anomaly determination unit 15, a time measurement unit 16, and a notification determination unit 17. The anomaly detection system 1000 according to embodiment 1 includes the anomaly detection device 1, an image capture device 2, and a notification unit 18. Note that the image capture device 2 and the notification unit 18 may be included in the anomaly detection device 1.

[0025] The imaging device 2 captures an image of the occupant. That is, the imaging device 2 acquires an image of the occupant. The image acquired by the imaging device 2 is output to the image acquisition unit 3. By including the imaging device 2, the abnormality detection system 1000 is able to capture an image of the occupant. Furthermore, by including the imaging device 2, the abnormality detection system 1000 is able to output an image of the occupant to the image acquisition unit 3.

[0026] The image acquisition unit 3 acquires a captured image of the occupant from the imaging device 2. The image acquisition unit 3 outputs the captured image to the face information detection unit 4 and the line of sight information detection unit 6.

[0027] The face information detection unit 4 acquires the captured image from the image acquisition unit 3. The face information detection unit 4 detects face direction information from the captured image. The face direction information includes facial feature information and facial feature reliability. The face information detection unit 4 outputs the face direction information to the face direction angle calculation unit 5.

[0028] The facial feature information includes, for example, positional relationship information of the eyes, nose, mouth, etc. The facial feature reliability is expressed, for example, as a value from 0 to 100. The facial feature reliability takes a smaller value when a part of the eyes, nose, mouth, etc. is occluded, for example, by covering the mouth with a hand, rubbing the eyes with a hand, or turning around from the driver's seat to look into the back seat, compared to other cases.

[0029] The face information detection unit 4 may detect, from the captured image, a face frame indicating the range of the detected face, a face detection reliability indicating the detection reliability of the face frame, and the like.

[0030] The face direction angle calculation unit 5 calculates the face direction angle of the occupant from positional relationship information of the eyes, nose, mouth, etc. That is, the face direction angle calculation unit 5 calculates the face direction angle from facial feature information. That is, the face direction angle calculation unit 5 calculates the face direction angle from face direction information. The face direction angle includes, for example, a pitch angle and a yaw angle.

[0031] The face direction angle calculation unit 5 outputs the calculated face direction angle and facial feature reliability to the face direction range determination unit 10. Furthermore, the face direction angle calculation unit 5 may output the calculated face direction angle to the gaze angle calculation unit 7. Furthermore, the face direction angle calculation unit 5 may output the calculated face direction angle to the determination angle calculation unit 14.

[0032] The gaze information detection unit 6 acquires the captured image from the image acquisition unit 3. The gaze information detection unit 6 detects gaze information from the captured image. The gaze information detection unit 6 outputs the gaze information to the gaze angle calculation unit 7. The gaze information detection unit 6 may also output the gaze information to the effective gaze determination unit 8.

[0033] The gaze information includes gaze reliability. The gaze reliability is expressed, for example, as a value between 0 and 100. For example, when an occupant is wearing sunglasses or the like that tend to block the light of the infrared sensor, the gaze reliability becomes smaller than in other cases because the contrast ratio around the eyes is reduced. Furthermore, the gaze reliability becomes smaller than in other cases because, for example, when the eyelids are drooped, the iris and pupil cannot be captured. Furthermore, the gaze reliability becomes smaller than in other cases because, for example, when the face direction angle with respect to the image capture device 2 is greater than a certain value, the contrast ratio of the iris and pupil is reduced due to the shadow of the eyelashes or eyelids, etc.

[0034] The gaze angle calculation unit 7 calculates the gaze angle of the occupant from the gaze information. The gaze information includes, for example, a right eye gaze angle and a left eye gaze angle. The gaze angle calculation unit 7 calculates the gaze angle as the gaze angle of both eyes using, for example, the right eye gaze angle, the left eye gaze angle, and the face direction angle acquired from the face direction angle calculation unit 5. The gaze angle includes, for example, a pitch angle and a yaw angle, and indicates the gaze direction of both eyes of the occupant.

[0035] The gaze angle calculation unit 7 outputs the calculated gaze angle and gaze reliability to the valid gaze determination unit 8. In addition, the gaze angle calculation unit 7 can output the face direction angle acquired from the face direction angle calculation unit 5 to the valid gaze determination unit 8.

[0036] Note that gaze angle calculation unit 7 may acquire gaze information and a determination result as to whether or not the gaze is valid from valid gaze determination unit 8. Furthermore, gaze angle calculation unit 7 may calculate the gaze angle when the gaze is valid based on the determination result acquired from valid gaze determination unit 8, and may not calculate the gaze angle when the gaze is not valid.

[0037] The valid gaze determination unit 8 determines whether the gaze of the occupant is valid or not using the gaze reliability acquired from the gaze angle calculation unit 7. The valid gaze determination unit 8 determines that the gaze is valid when the gaze reliability is equal to or greater than a predetermined threshold. The valid gaze determination unit 8 determines that the gaze is not valid when the gaze reliability is less than the predetermined threshold.

[0038] The valid gaze determination unit 8 outputs the gaze angle and the determination result as to whether the gaze is valid to the gaze range determination unit 9 and the face direction range determination unit 10. The valid gaze determination unit 8 can also output the face direction angle acquired from the gaze angle calculation unit 7 to the gaze range determination unit 9.

[0039] Note that the valid gaze determination unit 8 may acquire the gaze reliability directly from the gaze information detection unit 6 and determine whether the gaze is valid. When the valid gaze determination unit 8 acquires the gaze reliability from the gaze information detection unit 6 and determines that the gaze is not valid, the valid gaze determination unit 8 may output the determination result to the gaze angle calculation unit 7, and the gaze angle calculation unit 7 may omit the process of specifically calculating the gaze angle. In other words, the valid gaze determination unit 8 can determine that the gaze is not valid even when the gaze angle calculation unit 7 has not calculated a specific gaze angle.

[0040] Gaze range determination unit 9 acquires the gaze angle and the determination result as to whether or not the gaze is valid from valid gaze determination unit 8. If the gaze is valid, gaze range determination unit 9 determines whether or not the gaze angle calculated from the gaze information is within a predetermined gaze range.

[0041] The gaze range determination unit 9 compares the gaze angle calculated from the gaze information with the gaze range and determines whether the gaze angle calculated from the gaze information is within a predetermined gaze range. If the gaze angle calculated from the gaze information is within the predetermined gaze range, the gaze range determination unit 9 outputs the face direction angle acquired from the effective gaze determination unit 8 to the storage unit 11.

[0042] The line-of-sight range is defined as a range of normal line-of-sight angles. The line-of-sight range includes, for example, a pitch angle and a yaw angle. A line-of-sight range is defined for each of the pitch angle and the yaw angle. For example, upper and lower limits are defined for each of the pitch angle and the yaw angle. The range between the defined lower limit and the defined upper limit is the angle within the line-of-sight range.

[0043] The line-of-sight range is determined, for example, for each model of vehicle 100. That is, the line-of-sight range is determined, for example, as a value that does not change depending on the seat position. The line-of-sight range is determined in advance, for example, using the values ​​of the line-of-sight angles of multiple occupants with different physiques. That is, the line-of-sight range can be determined using non-abnormal line-of-sight angles measured at different seat positions, including the front-most and rear-most. In order to prevent false abnormality detection, it is desirable to set the line-of-sight range to a range that does not include an abnormal line-of-sight angle when the seat position is the front-most, and does not include an abnormal line-of-sight angle when the seat position is the rear-most.

[0044] The line-of-sight range is set to include the windshield, for example, and is a range defined by setting upper and lower limit values ​​for the pitch angle and yaw angle, respectively. That is, the line-of-sight range is set so that the line-of-sight range determiner 9 determines that the line-of-sight angle is within the line-of-sight range when the occupant's line of sight is directed in the direction in which the windshield is located, and determines that the line-of-sight angle is outside the line-of-sight range when the occupant's line of sight is directed to an area above, below, or to the left or right of the windshield.

[0045] The line of sight range determination unit 9 can determine whether the yaw angle included in the line of sight angle is within the line of sight range. That is, the line of sight range determination unit 9 can determine whether the occupant's line of sight is facing sideways. That is, the line of sight range determination unit 9 can output the facial direction angle when the occupant's line of sight is not facing sideways to the storage unit 11 as a normal facial direction angle. That is, the line of sight range determination unit 9 can not determine the facial direction angle when the occupant's line of sight is facing sideways as a normal facial direction angle.

[0046] The line-of-sight range determination unit 9 can determine whether the pitch angle included in the line-of-sight angle is within the line-of-sight range. That is, the line-of-sight range determination unit 9 can determine whether the occupant's line of sight is directed upward or downward. That is, the line-of-sight range determination unit 9 can output the face direction angle when the occupant's line of sight is not directed upward or downward to the storage unit 11 as the normal face direction angle. That is, the line-of-sight range determination unit 9 can not determine the face direction angle when the occupant's line of sight is directed upward or downward as the normal face direction angle. That is, the line-of-sight range determination unit 9 can not determine the face direction angle when the occupant's face is directed toward their knees, feet, or the like as the normal face direction angle.

[0047] It is desirable to determine whether the pitch angle and yaw angle are within the line of sight range, and if both the pitch angle and yaw angle are within the line of sight range, it is desirable to store the calculated face direction angle in the storage unit 11.

[0048] The upper and lower limits may be set for only the pitch angle or the yaw angle. Also, it may be possible to determine whether the line-of-sight angle is within the line-of-sight range for only the pitch angle or the yaw angle.

[0049] If the gaze is not valid, gaze range determination unit 9 may not output the facial direction angle acquired from valid gaze determination unit 8 to storage unit 11, but may output the determination result as to whether the gaze angle calculated from the gaze information is within a predetermined gaze range to storage unit 11. Furthermore, gaze range determination unit 9 may output the facial direction angle acquired from valid gaze determination unit 8 to abnormality determination unit 15.

[0050] The face direction range determination unit 10 acquires the face direction angle and the facial feature reliability from the face direction angle calculation unit 5. The face direction range determination unit 10 also acquires the determination result of whether the gaze is valid or not from the valid gaze determination unit 8.

[0051] When the line of sight is not valid, the face direction range determination unit 10 determines whether the face direction angle calculated from the face direction information is within a predetermined face direction range.

[0052] The face direction range determination unit 10 compares the face direction angle calculated from the face direction information with the face direction range, and determines whether the face direction angle calculated from the face direction information is within a predetermined face direction range. If the face direction angle calculated from the face direction information is within the predetermined face direction range, the face direction range determination unit 10 outputs the face direction angle calculated from the face direction information to the storage unit 11.

[0053] The face direction range is determined as a range of normal face direction angles. The face direction range includes, for example, a pitch angle and a yaw angle. A face direction range is set for each of the pitch angle and the yaw angle. For example, an upper limit and a lower limit are set for each of the pitch angle and the yaw angle. An angle within the face direction range is equal to or greater than the set lower limit and equal to or less than the set upper limit.

[0054] The face direction range is set to include, for example, the windshield, and is a range defined by setting upper and lower limit values ​​for the pitch angle and yaw angle, respectively. That is, the face direction range is set so that the face direction range determiner 10 determines that the face direction angle is within the face direction range when the occupant's face is facing the direction in which the windshield is located, and determines that the face direction angle is outside the face direction range when the occupant's face is facing an area above, below, or to the left or right of the windshield.

[0055] The face direction range determination unit 10 can determine whether the yaw angle included in the face direction angle is within the face direction range. That is, the face direction range determination unit 10 can determine whether the occupant's face is facing sideways. That is, the face direction range determination unit 10 can output the face direction angle when the occupant's face is not facing sideways to the storage unit 11 as a normal face direction angle. That is, the face direction range determination unit 10 can not determine the face direction angle when the occupant's face is facing sideways as a normal face direction angle.

[0056] The face direction range determination unit 10 can determine whether a pitch angle included in the face direction angle is within a face direction range. That is, the face direction range determination unit 10 can determine whether the occupant's face is facing up or down. That is, the face direction range determination unit 10 can output a face direction angle when the occupant's face is not facing up or down to the storage unit 11 as a normal face direction angle. That is, the face direction range determination unit 10 can not determine a face direction angle when the occupant's line of sight is facing up or down as a normal face direction angle. That is, the face direction range determination unit 10 can not determine a face direction angle when the occupant's face is facing their knees, feet, etc. as a normal face direction angle.

[0057] It is desirable to determine whether the pitch angle and yaw angle are within the respective face direction ranges, and if both the pitch angle and yaw angle are within the face direction ranges, it is desirable to store the face direction angle in the storage unit 11.

[0058] The upper and lower limits may be set for only the pitch angle or the yaw angle. Also, it may be possible to determine whether the face direction angle is within the face direction range for only the pitch angle or the yaw angle.

[0059] The face direction range is determined, for example, for each model of vehicle 100. That is, the face direction range is determined, for example, as a value that does not change depending on the seat position. The face direction range is determined in advance, for example, using the values ​​of the face direction angles of multiple occupants with different physiques. That is, the face direction range can be determined using face direction angles that are not abnormal measured at different seat positions including the front-most and rear-most. In order to prevent false detection of an abnormality, it is desirable to set the face direction range to a range that does not include an abnormal face direction angle when the seat position is the front-most, and does not include an abnormal face direction angle when the seat position is the rear-most.

[0060] The face direction range may be determined to be a range different from the line of sight range, or may be determined to be the same range as the line of sight range.

[0061] Note that even when the face direction angle calculated from the face direction information is not within a predetermined face direction range, face direction range determination unit 10 may output the face direction angle calculated from the face direction information to storage unit 11. When a face direction angle that is not within the predetermined face direction range is stored in storage unit 11, it is stored separately from a face direction angle that is within the predetermined face direction range.

[0062] If the facial feature reliability is less than a certain value, face direction range determination unit 10 may determine that the face direction angle calculated from the face direction information is not within a predetermined face direction range. Furthermore, if face direction range determination unit 10 determines that the face direction angle calculated from the face direction information is not within a predetermined face direction range, it may not be necessary to output the face direction angle to storage unit 11.

[0063] When the gaze angle calculated from the gaze information is within a predetermined gaze range, the storage unit 11 acquires the face direction angle from the gaze range determination unit 9. When the gaze angle calculated from the face direction information is within a predetermined face direction range, the storage unit 11 acquires the face direction angle from the face direction range determination unit 10.

[0064] Furthermore, when the gaze angle calculated from the gaze information is within a predetermined gaze range, storage unit 11 may acquire the face direction angle acquired when the gaze angle is within the predetermined gaze range from face direction range determination unit 10. That is, when the gaze is valid, storage unit 11 may acquire the face direction angle from face direction range determination unit 10 or may acquire the face direction angle from gaze range determination unit 9.

[0065] The storage unit 11 stores the facial direction angle acquired from the gaze range determination unit 9. That is, when the gaze is valid, the calculated facial direction angle is stored when the gaze angle is within a predetermined gaze range. That is, the calculated facial direction angle is stored in the storage unit 11 when the gaze angle is valid and within the predetermined gaze range. By including the storage unit 11, the anomaly detection device 1 can store the calculated facial direction angle when the gaze angle is within the predetermined gaze range.

[0066] When the line of sight is valid, the facial direction angle calculated when the line of sight angle is within a predetermined line of sight range is set as the first correction angle. Because the line of sight range is set as a range of normal line of sight angles, the first correction angle, which is the facial direction angle calculated when the line of sight angle is within the predetermined line of sight range, is stored as the normal facial direction angle in the memory unit 11. Furthermore, because the first correction angle is calculated from a captured image of the occupant, it is the normal facial direction angle for the seat position at the time of image capture.

[0067] The storage unit 11 may acquire the calculated face direction angle when the gaze angle is not within a predetermined gaze range, and store the calculated face direction angle separately from the first correction angle.

[0068] The storage unit 11 stores the face direction angle acquired from the face direction range determination unit 10. That is, when the line of sight is invalid, the storage unit 11 stores the face direction angle calculated when the face direction angle is within a predetermined face direction range. By including the storage unit 11, the abnormality detection device 1 can store the face direction angle acquired when the line of sight is invalid and the face direction angle is within a predetermined face direction range.

[0069] If the line of sight is not valid, the face direction angle calculated when the face direction angle is within a predetermined face direction range is set as the second correction angle. Because the face direction range is set as a normal face direction angle, the second correction angle is stored in the memory unit 11 as a normal face direction angle. In addition, because the second correction angle is calculated from a captured image of the occupant, it is the normal face direction angle for the seat position at the time of capturing the image.

[0070] The first correction angle and the second correction angle are referred to as corrective face direction angles. That is, the corrective face direction angles include the first correction angle and the second correction angle.

[0071] The storage unit 11 may acquire the calculated face direction angle when the face direction angle is not within a predetermined face direction range, and store the calculated face direction angle separately from the corrective face direction angle.

[0072] The storage unit 11 may store information other than the face direction angle acquired from the captured image. For example, the storage unit 11 may store gaze reliability, a determination result of whether the gaze is valid, facial feature reliability, etc.

[0073] The frequency distribution creation unit 12 creates a frequency distribution for face direction angles by referring to the face direction angles stored in the storage unit 11. That is, the frequency distribution creation unit 12 creates a frequency distribution for corrective face direction angles stored as normal face direction angles. For example, a frequency distribution is created for each of the pitch angle and yaw angle of the corrective face direction angles. Note that a frequency distribution may be created for only either the pitch angle or the yaw angle.

[0074] The frequency distribution creating unit 12 creates a frequency distribution when the number of corrective face direction angles stored in the memory unit 11 is equal to or greater than a predetermined number. By including the frequency distribution creating unit 12, the abnormality detection device 1 can create a frequency distribution from a plurality of corrective face direction angles.

[0075] The frontal face direction angle calculation unit 13 calculates the frontal face direction angle by using the frequency distribution created by the frequency distribution creation unit 12. For example, the frontal face direction angle calculation unit 13 sets the median of the most frequent class in the frequency distribution created by the frequency distribution creation unit 12 as the frontal face direction angle. The frontal face direction angle calculation unit 13 outputs the calculated frontal face direction angle to the determination angle calculation unit 14. By including the frontal face direction angle calculation unit 13, the abnormality detection device 1 can calculate the frontal face direction angle by using the frequency distribution created by the frequency distribution creation unit 12.

[0076] The forward facing angle includes, for example, a pitch angle and a yaw angle. The forward facing angle is the traveling direction of the vehicle 100, and is the direction in which the occupant faces the windshield 102.

[0077] The determination angle calculation unit 14 calculates a determination face direction angle used to determine whether the state of the occupant is abnormal. The determination angle calculation unit 14 calculates the determination face direction angle using the face direction angle calculated when the gaze angle is within a predetermined gaze range. The determination angle calculation unit 14 also calculates the determination face direction angle using the face direction angle calculated when the face direction angle is within a predetermined face direction range. The determination angle calculation unit 14 outputs the calculated determination face direction angle to the abnormality determination unit 15.

[0078] The determination angle calculation unit 14 stores a predetermined first correction value. Alternatively, the determination angle calculation unit 14 can obtain the first correction value from an external source. The first correction value is determined for each vehicle model of the vehicle 100, for example. In other words, the first correction value is determined as a value that does not change depending on the seat position, for example.

[0079] The first correction value can be set as the angle between a line segment extending from the occupant in a forward direction and a line segment extending from the occupant in a direction toward the image capture device 2. The positive direction of the first correction value can be set to a direction opposite to the positive direction of the face direction angle calculated from the face direction information. In other words, the first correction value can be determined so that when the occupant's face is facing forward, the sum of the first correction value and the face direction angle with respect to the image capture device 2 becomes 0°. The first correction value is, for example, an angle at the seat position when an occupant with an average build is riding in the vehicle. The first correction value includes, for example, a pitch angle and a yaw angle.

[0080] The determination angle calculation unit 14 calculates a second correction value using the first correction value and the front face direction angle acquired from the front face direction angle calculation unit 13. The second correction value is an angle calculated by adding the first correction value to the front face direction angle. That is, the second correction value is calculated using the front face direction angle calculated from the correction face direction angle including the first correction angle and the second correction angle. The second correction value includes, for example, a pitch angle and a yaw angle.

[0081] The determination angle calculation unit 14 calculates a determination face direction angle using the face direction angle calculated from the face direction information, the first correction value, and the second correction value. The determination face direction angle is an angle calculated by (the face direction angle calculated from the face direction information) - (the second correction value) + (the first correction value). The determination face direction angle includes, for example, a pitch angle and a yaw angle.

[0082] Note that if the number of corrective face direction angles stored in the storage unit 11 is less than a predetermined number and therefore a frequency distribution has not been created, the judgment angle calculation unit 14 does not calculate the second correction value. In other words, if a frequency distribution has not been created, the judgment angle calculation unit 14 calculates the judgment face direction angle without using the second correction value. If a frequency distribution has not been created, the judgment face direction angle is calculated as (face direction angle calculated from face direction information) + (first correction value).

[0083] The abnormality determination unit 15 determines whether the state of the occupant is abnormal using the determination face direction angle acquired from the determination angle calculation unit 14. The abnormality determination unit 15 determines whether the determination face direction angle is within a predetermined normal determination range. If the determination face direction angle is within the predetermined normal determination range, the abnormality determination unit 15 determines that the state of the occupant is not abnormal. Furthermore, if the determination face direction angle is not within the predetermined normal determination range, the abnormality determination unit 15 determines that the state of the occupant is abnormal.

[0084] The normal determination range is determined as a range of normal angles for the face direction angle. Normal determination ranges are set for, for example, the pitch angle and the yaw angle. For example, upper and lower limit values ​​are set for each of the pitch angle and the yaw angle. An angle within the normal determination range is an angle that is equal to or greater than the set lower limit value and equal to or less than the set upper limit value.

[0085] The normal determination range is set to include, for example, the windshield, and is defined by setting upper and lower limit values ​​for the pitch angle and yaw angle, respectively. That is, the normal determination range is set so that the abnormality determination unit 15 determines that the state of the occupant is not abnormal when the occupant faces the direction in which the windshield is located, and determines that the state of the occupant is abnormal when the occupant faces an area above or below or to the left or right of the windshield.

[0086] The abnormality determination unit 15 can determine whether the yaw angle included in the determined face direction angle is within a normal determination range. That is, the abnormality determination unit 15 can determine that there is an abnormality when the yaw angle included in the determined face direction angle is not within the normal determination range. That is, the abnormality determination unit 15 can determine that there is an abnormality when the occupant's face is facing sideways.

[0087] Furthermore, the abnormality determination unit 15 can determine whether or not the pitch angle included in the determination face direction angle is within the normal determination range. That is, the abnormality determination unit 15 can determine that there is an abnormality when the pitch angle included in the determination face direction angle is not within the normal determination range. That is, the abnormality determination unit 15 can determine that there is an abnormality when the occupant's face is facing up or down. That is, it can also determine that there is an abnormality when the occupant's face is facing toward their knees, feet, or the like.

[0088] It is desirable to determine whether the state of the occupant is abnormal for each of the pitch angle and the yaw angle. It is also desirable that the abnormality determination unit 15 determines that the state of the occupant is not abnormal when both the pitch angle and the yaw angle are within the normal determination range.

[0089] The upper and lower limits may be set for only the pitch angle or the yaw angle. When determining whether the condition of the occupant is abnormal with respect to only the pitch angle or the yaw angle, the determination may be made only with respect to the pitch angle or the yaw angle.

[0090] The normal determination range is determined, for example, for each model of the vehicle 100. That is, the normal determination range is determined, for example, as a value that does not change depending on the seat position.

[0091] The normal determination range is set to be different from the gaze range and face direction range, for example. The gaze range and face direction range are used to determine the face direction angle to be stored in the storage unit 11.

[0092] Furthermore, it is desirable that the width of the gaze range be equal to or less than the normal judgment range. That is, it is desirable that (upper limit of gaze range) - (lower limit of gaze range) ≦ (upper limit of normal judgment range) - (lower limit of normal judgment range). By ensuring that the width of the gaze range is equal to or less than the normal judgment range, it is possible to prevent a face direction angle outside the normal judgment range from being stored in memory unit 11 as a normal face direction angle.

[0093] Furthermore, it is desirable that the upper limit value of the gaze range be equal to or less than the upper limit value of the normal judgment range. By making the upper limit value of the gaze range equal to or less than the upper limit value of the normal judgment range, it is possible to prevent a face direction angle greater than the upper limit value of the normal judgment range from being stored in storage unit 11 as a normal face direction angle. It is also desirable that the lower limit value of the gaze range be equal to or greater than the lower limit value of the normal judgment range. By making the lower limit value of the gaze range equal to or greater than the lower limit value of the normal judgment range, it is possible to prevent a face direction angle smaller than the lower limit value of the normal judgment range from being stored in storage unit 11 as a normal face direction angle. It is also desirable that the upper limit value of the gaze range be equal to or less than the upper limit value of the normal judgment range and that the lower limit value of the gaze range be equal to or greater than the lower limit value of the normal judgment range.

[0094] Furthermore, it is desirable that the width of the face direction range be equal to or less than the normal judgment range. That is, it is desirable that (upper limit of face direction range) - (lower limit of face direction range) ≦ (upper limit of normal judgment range) - (lower limit of normal judgment range). By ensuring that the width of the face direction range is equal to or less than the normal judgment range, it is possible to prevent a face direction angle outside the normal judgment range from being stored in storage unit 11 as a normal face direction angle.

[0095] Furthermore, it is desirable that the upper limit value of the face direction range be equal to or less than the upper limit value of the normal judgment range. By making the upper limit value of the face direction range equal to or less than the upper limit value of the normal judgment range, it is possible to prevent a face direction angle greater than the upper limit value of the normal judgment range from being stored in the storage unit 11 as a normal face direction angle. It is also desirable that the lower limit value of the face direction range be equal to or greater than the lower limit value of the normal judgment range. By making the lower limit value of the face direction range equal to or greater than the lower limit value of the normal judgment range, it is possible to prevent a face direction angle smaller than the lower limit value of the normal judgment range from being stored in the storage unit 11 as a normal face direction angle. It is also desirable that the upper limit value of the face direction range be equal to or less than the upper limit value of the normal judgment range and that the lower limit value of the face direction range be equal to or greater than the lower limit value of the normal judgment range.

[0096] The abnormality determination unit 15 outputs the determination result as to whether or not the determined face direction angle is within a predetermined normal determination range to the time measurement unit 16. The abnormality determination unit 15 may also output the determination result as to whether or not the determined face direction angle is within a predetermined normal determination range to the notification determination unit 17. It is desirable that the abnormality determination unit 15 performs the determination every time the imaging device 2 acquires a captured image, that is, for each frame.

[0097] The abnormality determination unit 15 can acquire face direction information from the face information detection unit 4. When the facial feature reliability included in the face direction information acquired from the face information detection unit 4 is smaller than a threshold, the abnormality determination unit 15 can determine that the state of the occupant cannot be identified.

[0098] The time measurement unit 16 measures the time during which the determination that the occupant's condition is abnormal continues. By including the time measurement unit 16, the abnormality detection device 1 can measure the time during which the determination that the occupant's condition is abnormal continues.

[0099] When the time measurement unit 16 receives a determination that the occupant's condition is abnormal from the abnormality determination unit 15, the time measurement unit 16 increases the value of the determination counter from its initial value, i.e., updates the time during which the determination that the occupant's condition is abnormal continues.

[0100] If the value of the determination counter is greater than the initial value and the determination result obtained from the abnormality determination unit 15 is that there is no abnormality, the time measurement unit 16 increases the value of the reset counter from the initial value. In other words, it updates the time during which it has been determined that the state of the occupant is not abnormal.

[0101] When the determination result acquired by the time measurement unit 16 from the abnormality determination unit 15 indicates that the occupant's condition is not abnormal, if the value of the reset counter is equal to or greater than a predetermined reset counter threshold, the value of the determination counter is reset to its initial value. That is, if the determination that the occupant's condition is not abnormal continues for a certain period of time, the duration of the determination that the occupant is abnormal is reset. When the value of the determination counter is reset to its initial value, the value of the reset counter is also reset to its initial value.

[0102] In addition, when the value of the reset counter is greater than the initial value and the determination result acquired by the time measurement unit 16 from the abnormality determination unit 15 indicates an abnormality, the value of the reset counter is also returned to the initial value.

[0103] Furthermore, if the determination result obtained from the abnormality determination unit 15 indicates that the state of the occupant is unspecifiable, the value of the indefinite counter is increased from its initial value. That is, the time during which the determination that the state of the occupant is unspecifiable continues is updated.

[0104] In addition, if the value of the reset counter is greater than the initial value and the judgment result obtained by the time measurement unit 16 from the abnormality judgment unit 15 is that the condition of the occupant cannot be identified, the value of the reset counter is also returned to the initial value.

[0105] The method for measuring time is not limited to the above method.

[0106] The notification determination unit 17 determines whether or not to notify the occupant that the condition is abnormal based on the time measured by the time measurement unit 16. That is, it decides whether or not to notify. If the time during which the determination that the condition of the occupant is abnormal continues is equal to or longer than a predetermined reference time, the notification determination unit 17 determines to notify. If it determines to notify, the notification determination unit 17 outputs the determination result that a notification should be made to the notification unit 18.

[0107] If the time during which the determination that the occupant's condition is abnormal continues is shorter than a predetermined reference time, the notification determination unit 17 determines not to issue a notification. If it is determined not to issue a notification, the notification determination unit 17 does not notify the notification unit 18 of the determination result. Alternatively, if it is determined not to issue a notification, the notification determination unit 17 notifies the notification unit 18 of the determination result that no notification will be issued.

[0108] The notification determination unit 17 determines whether to issue a notification by, for example, referring to the value of a determination counter. If the value of the determination counter is equal to or greater than a predetermined continuation determination threshold, the notification determination unit 17 determines to issue a notification. If it determines to issue a notification, the notification determination unit 17 notifies the notification unit 18 of the determination result that a notification should be issued.

[0109] If the value of the determination counter is less than a predetermined continuation determination threshold, the notification determination unit 17 determines not to notify. If it is determined not to notify, the notification determination unit 17 does not notify the notification unit 18 of the determination result. Alternatively, if it is determined not to notify, the notification determination unit 17 notifies the notification unit 18 of the determination result that not to notify.

[0110] If the value of the indefinite counter is less than a predetermined continuous indefinite threshold, the notification determination unit 17 determines not to issue a notification. Furthermore, if the value of the indefinite counter is less than the predetermined continuous indefinite threshold, the notification determination unit 17 maintains the value of the determination counter. That is, it does not change the value of the determination counter. Furthermore, if the value of the indefinite counter is equal to or greater than the predetermined continuous indefinite threshold, it determines that the device is unidentifiable and resets the value of the determination counter to its initial value. If the value of the indefinite counter is equal to or greater than the predetermined continuous indefinite threshold, the notification determination unit 17 determines not to issue a notification. If it is determined not to issue a notification, the notification determination unit 17 does not notify the notification unit 18 of the determination result that no notification should be issued. Alternatively, if it is determined not to issue a notification, the notification determination unit 17 notifies the notification unit 18 of the determination result that no notification should be issued.

[0111] When the determination result acquired from the abnormality determination unit 15 indicates an abnormality, the notification determination unit 17 may immediately determine to issue a notification without referring to the time measurement unit 16, and notify the notification unit 18. It is desirable that the notification determination unit 17 perform the determination every time the imaging device 2 acquires an image, that is, for each frame.

[0112] The notification unit 18 is, for example, an audio device such as a speaker. The notification unit 18 may be provided in, for example, an audio device provided in the vehicle 100. The notification unit 18 notifies that the occupant's condition is abnormal by, for example, outputting a warning sound or a voice message. The notification unit 18 may also be, for example, a display device such as a display. The notification unit 18 notifies that the occupant's condition is abnormal by, for example, displaying characters or a mark. The notification unit 18 may also be a turn signal, hazard lamp, headlight, or the like provided on the exterior of the vehicle so as to be visible from other vehicles. The notification unit 18 notifies that the occupant's condition is abnormal by, for example, lighting up or flashing.

[0113] The notification unit 18 acquires a notification from the notification determination unit 17. When the notification unit 18 acquires a notification from the notification determination unit 17 that a notification is to be issued, the notification unit 18 notifies the occupant of the vehicle 100 that the condition of the occupant is abnormal. Furthermore, when the notification unit 18 acquires a notification from the notification determination unit 17 that a notification is to be issued, the notification unit 18 may notify a location outside the vehicle 100 that the condition of the occupant is abnormal. Furthermore, the notification unit 18 may be provided in a vehicle other than the vehicle 100 in which the occupant whose condition is to be determined is riding.

[0114] By including the notification unit 18, the abnormality detection system 1000 can notify the occupant of the vehicle 100 that the occupant's condition is abnormal. Furthermore, by including the notification unit 18, the abnormality detection system 1000 can notify the outside of the vehicle 100 that the occupant's condition is abnormal.

[0115] Next, an example of the hardware configuration of the anomaly detection device 1 will be described. Fig. 4 is a diagram showing an example of the hardware configuration of the anomaly detection device 1 according to embodiment 1. The image acquisition unit 3, face information detection unit 4, face direction angle calculation unit 5, gaze information detection unit 6, gaze angle calculation unit 7, effective gaze determination unit 8, gaze range determination unit 9, face direction range determination unit 10, memory unit 11, frequency distribution creation unit 12, front face direction angle calculation unit 13, determination angle calculation unit 14, anomaly determination unit 15, time measurement unit 16, and notification determination unit 17 included in the anomaly detection device 1 may be a processing circuit 200a which is dedicated hardware as shown in Fig. 4(a) or a processor 200b which executes a program stored in memory 200c as shown in Fig. 4(b).

[0116] As shown in FIG. 4A , when the image acquisition unit 3, the face information detection unit 4, the face direction angle calculation unit 5, the gaze information detection unit 6, the gaze angle calculation unit 7, the effective gaze determination unit 8, the gaze range determination unit 9, the face direction range determination unit 10, the memory unit 11, the frequency distribution creation unit 12, the front face direction angle calculation unit 13, the determination angle calculation unit 14, the abnormality determination unit 15, the time measurement unit 16, and the notification determination unit 17 are dedicated hardware, the processing circuit 200 a corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of each of the image acquisition unit 3, face information detection unit 4, face direction angle calculation unit 5, gaze information detection unit 6, gaze angle calculation unit 7, effective gaze determination unit 8, gaze range determination unit 9, face direction range determination unit 10, memory unit 11, frequency distribution creation unit 12, front face direction angle calculation unit 13, determination angle calculation unit 14, abnormality determination unit 15, time measurement unit 16, and alarm determination unit 17 may be realized individually by a processing circuit, or the functions of each unit may be realized together by a single processing circuit.

[0117] 4B, when the image acquisition unit 3, the face information detection unit 4, the face direction angle calculation unit 5, the gaze information detection unit 6, the gaze angle calculation unit 7, the effective gaze determination unit 8, the gaze range determination unit 9, the face direction range determination unit 10, the memory unit 11, the frequency distribution creation unit 12, the forward face direction angle calculation unit 13, the determination angle calculation unit 14, the abnormality determination unit 15, the time measurement unit 16, and the notification determination unit 17 are the processor 200b, the functions of each unit are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 200c. The processor 200b reads out and executes the programs stored in the memory 200c, thereby realizing the functions of the image acquisition unit 3, face information detection unit 4, face direction angle calculation unit 5, gaze information detection unit 6, gaze angle calculation unit 7, effective gaze determination unit 8, gaze range determination unit 9, face direction range determination unit 10, memory unit 11, frequency distribution creation unit 12, front face direction angle calculation unit 13, determination angle calculation unit 14, abnormality determination unit 15, time measurement unit 16, and alarm determination unit 17. That is, the image acquisition unit 3, the face information detection unit 4, the face direction angle calculation unit 5, the gaze information detection unit 6, the gaze angle calculation unit 7, the effective gaze determination unit 8, the gaze range determination unit 9, the face direction range determination unit 10, the memory unit 11, the frequency distribution creation unit 12, the front face direction angle calculation unit 13, the determination angle calculation unit 14, the abnormality determination unit 15, the time measurement unit 16, and the notification determination unit 17 each include a memory 200c for storing a program that, when executed by the processor 200b, results in the execution of each step shown in FIG. 5 described below. In addition, these programs can also be said to cause a computer to execute the procedures or methods of the image acquisition unit 3, face information detection unit 4, face direction angle calculation unit 5, gaze information detection unit 6, gaze angle calculation unit 7, effective gaze determination unit 8, gaze range determination unit 9, face direction range determination unit 10, memory unit 11, frequency distribution creation unit 12, front face direction angle calculation unit 13, determination angle calculation unit 14, abnormality determination unit 15, time measurement unit 16, and alarm determination unit 17.

[0118] Here, the processor 200b is, for example, a CPU (Central Processing Unit), a processing unit, an arithmetic unit, a processor, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor). The memory 200c may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically EPROM), a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a minidisc, a CD (Compact Disc), or a DVD (Digital Versatile Disc).

[0119] Note that the functions of the image acquisition unit 3, face information detection unit 4, face direction angle calculation unit 5, gaze information detection unit 6, gaze angle calculation unit 7, effective gaze determination unit 8, gaze range determination unit 9, face direction range determination unit 10, memory unit 11, frequency distribution creation unit 12, front face direction angle calculation unit 13, determination angle calculation unit 14, abnormality determination unit 15, time measurement unit 16, and notification determination unit 17 may be partially realized by dedicated hardware and partially realized by software or firmware. In this way, the processing circuit 200a in the abnormality detection device 1 can realize the above-mentioned functions by hardware, software, firmware, or a combination thereof.

[0120] Next, the operation of the anomaly detection device 1 will be described. Fig. 5 is a flowchart showing an example of the operation of the anomaly detection device 1 according to the first embodiment. The operation of the anomaly detection device 1 is started, for example, when the ignition of the vehicle 100 is turned on or when the seat belt is fastened. Furthermore, although the flowchart in Fig. 5 does not show a process for terminating the operation of the anomaly detection device 1, the anomaly detection device 1 can terminate its operation, for example, when the ignition of the vehicle 100 is turned off or when the seat belt is unfastened.

[0121] First, the image capturing device 2 captures an image of the occupant. That is, the image capturing device 2 acquires an image of the occupant. The image acquiring unit 3 acquires the image from the image capturing device 2 (ST101).

[0122] Next, the face information detection unit 4 detects face direction information from the captured image (ST102), and the line of sight information detection unit 6 detects line of sight information from the captured image (ST102).

[0123] Next, the face direction angle calculation unit 5 calculates the face direction angle of the occupant using the facial feature information included in the detected face direction information (ST103). Also, the gaze angle calculation unit 7 calculates the gaze angle of the occupant using the detected gaze information (ST103). Note that the calculation of the face direction angle and the gaze angle may be performed after step ST104 described below, or may be performed simultaneously with step ST104.

[0124] Next, the gaze range determination unit 9 determines whether the gaze is valid or not using the gaze reliability included in the gaze information (ST104). If the gaze range determination unit 9 determines that the gaze is valid (ST104; YES), the process proceeds to step ST105. That is, if the gaze is valid, the process proceeds to step ST105. If the gaze range determination unit 9 determines that the gaze is not valid (ST104; NO), the process proceeds to step ST106. That is, if the gaze is not valid, the process proceeds to step ST106.

[0125] If the line of sight is valid (ST104; YES), line of sight range determination unit 9 determines whether the line of sight angle calculated from the line of sight information is within a predetermined line of sight range (ST105). If the line of sight angle calculated from the line of sight information is within the predetermined line of sight range (ST105; YES), the process proceeds to step ST107.

[0126] If the gaze angle calculated from the gaze information is not within the predetermined gaze range (ST105; NO), the process returns to step ST101. Note that if the gaze angle calculated from the gaze information is not within the predetermined gaze range (ST105; NO), the process may proceed to step ST111.

[0127] If the line of sight is not valid (ST104; NO), the face direction range determination unit 10 determines whether the face direction angle detected from the face direction information is within a predetermined face direction range (ST106). If the face direction angle calculated from the face direction information is within the predetermined face direction range (ST106; YES), the process proceeds to step ST108.

[0128] If the face direction angle calculated from the face direction information is not within the predetermined face direction range (ST106; NO), the process returns to step ST101. Note that if the face direction angle calculated from the face direction information is not within the predetermined face direction range (ST106; NO), the process may proceed to step ST111.

[0129] If the gaze angle calculated from the gaze information is within the predetermined gaze range (ST105; YES), the storage unit 11 acquires and stores the face direction angle when the gaze angle calculated from the gaze information is within the predetermined gaze range (ST107). That is, the storage unit 11 stores the first correction angle.

[0130] If the face direction angle calculated from the face direction information is within a predetermined face direction range (ST106; YES), the storage unit 11 acquires and stores the face direction angle when the face direction angle calculated from the face direction information is within the predetermined face direction range (ST108). That is, the storage unit 11 stores the second correction angle.

[0131] The storage unit 11 stores the first correction angle and the second correction angle, that is, the storage unit 11 stores the corrective face direction angles.

[0132] Next, the frequency distribution creation unit 12 determines whether the number of corrective face direction angles is equal to or greater than a predetermined number (ST109). That is, the frequency distribution creation unit 12 determines whether the number of corrective face direction angles stored in the storage unit 11 is equal to or greater than a predetermined number. Note that even if the number of corrective face direction angles stored is equal to or greater than a certain number, the storage unit 11 repeatedly stores the corrective face direction angles.

[0133] If the number of corrective face direction angles is equal to or greater than a predetermined number (ST109; YES), the process proceeds to step ST110. If the number of corrective face direction angles is not equal to or greater than a predetermined number (ST109; NO), the process returns to step ST101. Note that if the number of corrective face direction angles is not equal to or greater than a predetermined number (ST109; NO), the process may proceed to step ST111.

[0134] If the number of corrective face direction angles is equal to or greater than a predetermined number (ST109; YES), frequency distribution creation unit 12 creates a frequency distribution for face direction angles using the corrective face direction angles stored in storage unit 11. Frontal face direction angle calculation unit 13 calculates a frontal face direction angle using the frequency distribution created by frequency distribution creation unit 12. Determination angle calculation unit 14 calculates a second correction value using the first correction value and the frontal face direction angle acquired from frontal face direction angle calculation unit 13 (ST110).

[0135] The determination angle calculation unit 14 calculates a determination face direction angle using the face direction angle calculated from the face direction information, the first correction value, and the second correction value (ST111). The determination angle calculation unit 14 outputs the calculated determination face direction angle to the abnormality determination unit 15.

[0136] If the gaze angle calculated from the gaze information is not within a predetermined gaze range (ST105; NO), if the face direction angle calculated from the face direction information is not within a predetermined face direction range (ST106; NO), or if the number of corrective face direction angles is not equal to or greater than a predetermined number (ST109; NO), and the process reaches step ST111 without performing the processing of step ST110, the determination angle calculation unit 14 can calculate the determination face direction angle without using the second correction value. In other words, if a frequency distribution has not been created, the determination angle calculation unit 14 can calculate the determination face direction angle without using the second correction value.

[0137] Next, the abnormality determination unit 15 determines whether the state of the occupant is abnormal using the determined face direction angle acquired from the determined angle calculation unit 14 (ST112). The abnormality determination unit 15 determines whether the determined face direction angle is within a predetermined normal determination range. If the determined face direction angle is not within the predetermined normal determination range, the abnormality determination unit 15 determines that the state of the occupant is abnormal (ST112; YES), and proceeds to the processing of step ST113.

[0138] If the determined face direction angle is within a predetermined normal determination range, the abnormality determination unit 15 determines that the occupant's condition is not abnormal (ST112; NO), and returns to the processing of step ST101. If the occupant's condition is not abnormal, for example, the time measurement unit 16 may perform processing to increase the value of the reset counter from its initial value.

[0139] If the occupant's condition is abnormal (ST112; YES), the timer 16 updates the time period during which the occupant's condition has been determined to be abnormal (ST113). If the occupant's condition is abnormal, for example, the timer 16 performs processing to increase the value of the determination counter from its initial value.

[0140] In step ST112, if the facial feature reliability included in the face direction information is smaller than the threshold, the abnormality determination unit 15 may determine that the state of the occupant cannot be identified. If the state of the occupant cannot be identified, the process may return to step ST101. Furthermore, if the state of the occupant cannot be identified, the time measurement unit 16 may, for example, perform a process of increasing the value of a reset counter from its initial value.

[0141] The notification determination unit 17 determines whether to issue a notification based on the time period during which the determination that the occupant's condition is abnormal continues (ST114). The notification determination unit 17 determines whether to issue a notification by, for example, referring to the value of a determination counter.

[0142] If the time during which the determination that the occupant's condition is abnormal continues is equal to or longer than the predetermined reference time, the notification determination unit 17 determines to issue a notification (ST114; YES). If the notification determination unit 17 determines to issue a notification, the process proceeds to step ST115.

[0143] If the time during which the determination that the occupant's condition is abnormal continues is shorter than the predetermined reference time, the notification determination unit 17 determines not to issue a notification (ST114; NO). If the notification determination unit 17 determines not to issue a notification, it returns to the processing of step ST101.

[0144] When the notification unit 18 receives a notification from the notification determination unit 17 that a notification should be made, the notification unit 18 notifies the driver that the occupant's condition is abnormal (ST115). After the notification unit 18 makes the notification, the process can return to step ST101. Thereafter, the processes from step ST101 to step ST115 can be repeated.

[0145] If the process of step ST112 is performed without performing the process of step ST110, the second correction value may be calculated while the processes of steps ST101 to ST115 are repeated.

[0146] The abnormality detection device 1 calculates a second correction value and uses the second correction value to calculate the judgment face direction angle, thereby reducing the possibility of an abnormality not being detected or an abnormality being erroneously detected due to the normal face direction angle relative to the imaging device 2 differing depending on the seat position.

[0147] Furthermore, when the line of sight is valid, the anomaly detection device 1 determines the corrective face direction angle used to calculate the second correction value using the line of sight range. When the line of sight is valid, the device determines whether the line of sight angle is within the line of sight range, rather than whether the face direction angle is within the face direction range. By using the face direction angle when the line of sight angle is within the line of sight range as the corrective face direction angle, the possibility of an anomaly not being detected or an anomaly being erroneously detected can be reduced more than before.

[0148] The facial direction angle and the gaze angle may be different angles. For example, even if the facial direction angle is normal, the gaze angle may not be normal. For example, the face may be facing forward, but the gaze may be directed toward the side mirror. If the gaze angle is not normal, it is possible that the occupant is looking aside, even if the facial direction angle is normal, so it is desirable to determine that the occupant's condition is abnormal.

[0149] By using the face direction angle when the gaze angle is within the gaze range as the correction face direction angle, even if the face direction is normal, if the gaze angle is not normal, the face direction angle will not be used as the correction face direction angle. In other words, the possibility of an abnormality not being detected can be reduced more than before.

[0150] Furthermore, even if the face direction angle is not a normal face direction angle, if the gaze angle is normal, the face direction angle can be used as a correction face direction angle. If the gaze angle is normal, it is desirable to determine that the occupant's condition is not abnormal, because there is a possibility that the occupant is not looking aside, even if the face direction angle is not normal.

[0151] By using the face direction angle when the gaze angle is within the gaze range as the correction face direction angle, even if the face direction is not a normal face direction angle, the face direction angle can be used as the correction face direction angle when the gaze angle is a normal gaze angle. In other words, the possibility of an abnormality being erroneously detected can be reduced more than before.

[0152] Furthermore, when the line of sight is valid, the abnormality detection device 1 determines whether the line of sight angle is within the line of sight range, and when the line of sight angle is within the line of sight range, the calculated face direction angle is used as the correction face direction angle, thereby reducing the possibility of an abnormality not being detected or an abnormality being erroneously detected more than before.

[0153] When the line of sight is invalid, the face direction angle to be used as the correction face direction angle can be determined by determining whether the face direction angle is within a face direction range. By performing the determination using the face direction range when the line of sight is invalid, the correction face direction angle for calculating the second correction value can be stored in the storage unit 11 even when the line of sight is invalid.

[0154] Embodiment 2 An anomaly detection device 1 according to embodiment 2 will be described with reference to FIG. 6. FIG. 6 is a flowchart showing an example of the operation of the anomaly detection device 1 according to embodiment 2. Description of the same configuration as in embodiment 1 will be omitted. Furthermore, in FIG. 6, the same reference numerals as in FIGS. 1 to 5 indicate the same or equivalent parts. Furthermore, steps that are the same as the processing of the anomaly detection device 1 according to embodiment 1 are assigned the same reference numerals as those shown in FIG. 5, and description thereof will be omitted or simplified. Below, differences from embodiment 1 will be mainly described.

[0155] If the gaze is valid (ST104; YES), gaze range determination unit 9 determines whether the gaze angle calculated from the gaze information is within a predetermined gaze range (ST105). If the gaze angle calculated from the gaze information is within the predetermined gaze range (ST105; YES), storage unit 11 acquires and stores the face direction angle when the gaze angle calculated from the gaze information is within the predetermined gaze range (ST107).

[0156] If the line of sight is valid (ST104; YES), the process can proceed to step ST201 after processing step ST107. The determination angle calculation unit 14 calculates a determination line of sight angle used to determine whether or not the state of the occupant is abnormal. The determination angle calculation unit 14 calculates a determination line of sight angle using the line of sight angle calculated from the line of sight information and the first correction value (ST201). The determination line of sight angle is calculated as (line of sight angle calculated from the line of sight information) + (first correction value). In other words, the determination angle calculation unit 14 calculates the determination line of sight angle without using the second correction value. The determination angle calculation unit 14 outputs the calculated determination line of sight angle to the abnormality determination unit 15.

[0157] The first correction value used in calculating the determination gaze angle may be the same as the first correction value used in calculating the determination face direction angle. Note that the determination angle calculation unit 14 may calculate the determination gaze angle using a value different from the first correction value.

[0158] In order to perform the process of step ST201, the judgment angle calculation unit 14 can acquire the gaze angle from the gaze angle calculation unit 7, the valid gaze determination unit 8, the gaze range determination unit 9, the storage unit 11, or the like. Furthermore, the valid gaze determination unit 8 can output the gaze angle and the determination result as to whether the gaze is valid or not to the judgment angle calculation unit 14, the abnormality determination unit 15, or the like.

[0159] Although the case where the process proceeds to step ST201 after the process of step ST107 has been described, the process of ST201 may be performed simultaneously with the process of ST105 and the process of ST107.

[0160] Next, if the line of sight is valid, abnormality determination unit 15 determines whether or not there is an abnormality using the determined line of sight angle acquired from determination angle calculation unit 14 (ST202). If the line of sight is valid, abnormality determination unit 15 determines whether or not the determined line of sight angle is within a predetermined normal determination range. If the determined line of sight angle is not within the predetermined normal determination range, abnormality determination unit 15 determines that the state of the occupant is abnormal (ST202; YES), and proceeds to the processing of step ST113.

[0161] If the determined gaze angle is within a predetermined normal determination range, it is determined that the occupant's condition is not abnormal (ST202; NO), and the process returns to step ST101. If the occupant's condition is not abnormal, for example, the time measurement unit 16 may perform a process of increasing the value of a reset counter from its initial value.

[0162] The normal determination range used in step ST202 can be set to the same range as the normal determination range used in step ST112.

[0163] The abnormality determination unit 15 can acquire a determination result as to whether or not the line of sight is valid from the valid line of sight determination unit 8 or the determination angle calculation unit 14, etc. If the line of sight is valid, the abnormality determination unit 15 can make a determination using the determination line of sight angle acquired from the determination angle calculation unit 14, and if the line of sight is not valid, the abnormality determination unit 15 can make a determination using the determination face direction angle acquired from the determination angle calculation unit 14.

[0164] When the line of sight is valid, the judgment angle calculation unit 14 calculates the judgment line of sight angle using the line of sight angle, and the abnormality determination unit 15 determines whether or not there is an abnormality using the judgment line of sight angle, so that it is possible to determine that the state of the occupant is abnormal if the line of sight angle is not a normal line of sight angle regardless of the face direction angle. Therefore, it is possible to reduce the possibility of an abnormality not being detected or an abnormality being erroneously detected compared to conventional methods.

[0165] The abnormality detection system 1000 is equipped with a gaze range determination unit that determines whether the gaze angle is within a predetermined gaze range when the gaze is valid, and a determination angle calculation unit that calculates a determined face direction angle using the calculated face direction angle when the gaze is valid and the gaze angle is within the predetermined gaze range, thereby reducing the possibility of an abnormality not being detected or an abnormality being erroneously detected more than before.

[0166] The abnormality detection method can include steps ST101 to ST115, step ST201, and step ST202. The abnormality detection method includes a step in which gaze range determination unit 9 determines whether the gaze angle is within a predetermined gaze determination range when the gaze is valid, and a step in which determination angle calculation unit 14 calculates a determination face direction angle using the calculated face direction angle when the gaze angle is within the predetermined gaze determination range, thereby making it possible to reduce the possibility of an abnormality not being detected or an abnormality being erroneously detected more than before.

[0167] The program is stored in a readable storage medium, for example, a non-volatile or volatile semiconductor memory such as a RAM, ROM, flash memory, EPROM, or EEPROM, a magnetic disk such as a hard disk or flexible disk, or an optical disk such as a minidisk, CD, or DVD, and by having a computer execute the steps of: a gaze range determination unit determining whether the gaze angle is within a predetermined gaze determination range when the gaze is valid; and a determination angle calculation unit calculating a determined face direction angle using the calculated face direction angle when the gaze angle is within the predetermined gaze determination range, the possibility of an abnormality not being detected or an abnormality being falsely detected can be reduced more than before.

[0168] In the above-described embodiments, the materials, materials, dimensions, shapes, relative positional relationships, and implementation conditions of each component may be described. However, these are merely examples in all respects and are not intended to limit the scope of each embodiment. Therefore, countless variations not exemplified are contemplated within the scope of each embodiment. For example, these include cases where any component is modified, added, or omitted, and even cases where at least one component in at least one embodiment is extracted and combined with a component in another embodiment.

[0169] It goes without saying that various design modifications are possible within the scope of the present invention, as long as the object of the present invention can be achieved and the gist of the present invention is not deviated from.

[0170] DESCRIPTION OF SYMBOLS 1 Anomaly detection device, 2 Imaging device, 3 Image acquisition unit, 4 Face information detection unit, 5 Face direction angle calculation unit, 6 Gaze information detection unit, 7 Gaze angle calculation unit, 8 Effective gaze determination unit, 9 Gaze range determination unit, 10 Face direction range determination unit, 11 Memory unit, 12 Frequency distribution creation unit, 13 Front face direction angle calculation unit, 14 Determination angle calculation unit, 15 Anomaly determination unit, 16 Time measurement unit, 17 Notification determination unit, 18 Notification unit, 1000 Anomaly detection system

Claims

1. An abnormality detection device comprising: an image acquisition unit that acquires an image of an occupant; a facial direction angle calculation unit that calculates a facial direction angle of the occupant from the captured image; a gaze angle calculation unit that calculates a gaze angle of the occupant from the captured image; a gaze range determination unit that determines whether the gaze angle is within a predetermined gaze range; a judgment angle calculation unit that calculates a judgment facial direction angle used to determine whether or not the condition of the occupant is abnormal using the calculated facial direction angle when the gaze angle is within the predetermined gaze range; and an abnormality determination unit that determines that the condition of the occupant is abnormal when the judgment facial direction angle is not within a predetermined normal judgment range.

2. The abnormality detection device of claim 1, wherein the judgment angle calculation unit uses the gaze angle to calculate a judgment gaze angle used to judge whether the occupant's condition is abnormal or not, and the abnormality judgment unit judges that the occupant's condition is abnormal if the judgment gaze angle is not within the predetermined normal judgment range when the occupant's gaze is valid.

3. An abnormality detection device as described in claim 1 or 2, further comprising: a valid gaze determination unit that determines whether the occupant's gaze is valid or not; and a facial direction range determination unit that determines whether the facial direction angle is within a predetermined facial direction range when the gaze is not valid, wherein the determination angle calculation unit calculates the determined facial direction angle using the facial direction angle calculated when the facial direction angle is within the predetermined facial direction range.

4. An abnormality detection device as described in any one of claims 1 to 3, further comprising a memory unit that stores the calculated facial direction angle when the occupant's line of sight is valid and the line of sight angle is within a predetermined line of sight range.

5. The anomaly detection device according to claim 4, wherein the memory unit stores the calculated facial direction angle when the line of sight is invalid and the facial direction angle is within a predetermined facial direction range.

6. An anomaly detection device as described in any one of claims 1 to 5, further comprising a frequency distribution creation unit that creates a frequency distribution for the facial direction angle calculated when the gaze angle is within the predetermined gaze range.

7. An abnormality detection device as described in any one of claims 1 to 6, further comprising: a time measurement unit that measures the time during which the determination that the occupant's condition is abnormal continues; and an alarm determination unit that decides to issue an alarm if the time during which the occupant's condition is abnormal continues as measured by the time measurement unit is equal to or longer than a predetermined reference time.

8. An abnormality detection system comprising: an abnormality detection device according to any one of claims 1 to 7; and an imaging device that images the occupant and outputs the image to the image acquisition unit.

9. The abnormality detection system according to claim 8, further comprising an alarm unit that, when the abnormality determination unit determines that the condition of the occupant is abnormal, issues an alarm that the condition of the occupant is abnormal.

10. An abnormality detection method comprising the steps of: an image acquisition unit acquiring an image of an occupant; a facial direction angle calculation unit calculating the facial direction angle of the occupant from the captured image; a gaze angle calculation unit calculating the gaze angle of the occupant from the captured image; a gaze range determination unit determining whether the gaze angle is within a predetermined gaze determination range; a determination angle calculation unit calculating a determination facial direction angle used for determining whether the condition of the occupant is abnormal or not, using the facial direction angle calculated when the gaze angle is within the predetermined gaze determination range; and an abnormality determination unit determining that the condition of the occupant is abnormal when the determination facial direction angle is not within the predetermined normal determination range.

11. A program causing a computer to execute the steps of: an image acquisition unit acquiring an image of an occupant; a facial direction angle calculation unit calculating the facial direction angle of the occupant from the captured image; a gaze angle calculation unit calculating the gaze angle of the occupant from the captured image; a gaze range determination unit determining whether the gaze angle is within a predetermined gaze determination range; a determination angle calculation unit calculating a determination facial direction angle used for determining whether the condition of the occupant is abnormal or not, using the facial direction angle calculated when the gaze angle is within the predetermined gaze determination range; and an abnormality determination unit determining that the condition of the occupant is abnormal if the determination facial direction angle is not within the predetermined normal determination range.

Images