OCCUPANT CONDITION DETERMINATION DEVICE, OCCUPANT CONDITION DETERMINATION SYSTEM, OCCUPANT CONDITION DETERMINATION METHOD AND PROGRAM

Abstract

An occupant state determination device (1) of the present disclosure includes a face detection unit (12), a feature detection unit (13), a reference posture estimation unit (14), a region determination unit (151) that performs region determination as to whether at least one of a face orientation and a head position of the occupant or a displacement amount of the at least one of the face orientation and the head position of the occupant from a reference posture is one of a first region, a second region, and a third region set in a real space, a posture collapse determination unit (152) that performs posture collapse type determination, a face change degree detection unit (153) that detects a change degree of the occupant's face, an error detection determination unit (154),which determines whether the posture of the occupant is a posture collapse type based on the detection result of the facial change degree detection unit (153), and a problem posture determination unit (155) which determines whether a problem posture of the driver exists based on at least one of the area determination result, the posture collapse type determination result, and the detection error determination result.

Description

Technical field

[0001] The present disclosure relates to an occupant condition determining apparatus, an occupant condition determining system, an occupant condition determining method, and a program. General state of the art

[0002] A technique is developed for taking a picture of the interior of a vehicle, determining a posture error and detecting whether a problem such as a driver loss of consciousness exists based on a face orientation, head position, or the like of the occupant extracted from the image capture, and preventing a false detection of a posture error in which it is determined that a posture error exists when there is no posture error based on possible actions of the occupant without a posture error.

[0003] For example, a driver monitoring system of Patent Document 1 includes a problem diagnosis unit that determines that a driver problem exists when an absolute value of a roll angle of a driver's facial orientation is maintained at or above a predetermined angle (20°, preferably 30°, more preferably 40°) for at least a predetermined period of time (ten seconds or several tens of seconds), and a change degree detection unit that detects a change degree of the driver's face based on facial images. In the case where the change degree of the driver detected by the change degree detection unit is at or above a predetermined reference (for example, the driver's blinking frequency per unit time is at or above a reference frequency (for example, 1 time)), despite a determination by the problem diagnosis unit that a driver problem exists,that there is no problem with the driver. Prior art documentsPatent documents

[0004] Patent document 1 JP 2019-087018 A Summary of the inventionObject of the present invention

[0005] However, actions that an occupant can perform when not in a posture disorder can occur even when a posture disorder actually exists. Examples of actions that an occupant can perform when not in a posture disorder can occur when a posture disorder occurs during blinking, or when a posture disorder occurs during blinking and the blinking continues even after the occupant has assumed the posture disorder. In such a case, since the driver monitoring system detects blinking as an action that an occupant can perform when not in a posture disorder, it may determine that the occupant does not have a posture disorder and therefore does not have a problem, even though a posture disorder actually exists.Therefore, there is a problem that despite the presence of a malposture, this malposture is not detected due to the determination that there is no decrease in force, and thus the detection accuracy of the occupant condition is reduced.

[0006] The invention has been made to solve this problem, and its object is to provide an occupant state determining device with increased detection accuracy of the occupant state. Means of solving the task

[0007] The occupant state determining device of the present disclosure includes a face detection unit that detects the face of the occupant in an image in which the face of an occupant of a vehicle is captured, a feature detection unit that detects at least one of a face orientation and a head position of the occupant and face parts based on the face of the occupant detected by the face detection unit in the image, a reference posture estimation unit that estimates at least one of the face orientation and the head position of the occupant, which are used as a reference posture, based on the at least one of the face orientation and the head position of the occupant detected by the feature detection unit within a predetermined period of time, an area determination unit that performs an area determination therefor,whether the at least one of the face orientation and the head position of the occupant detected by the feature detection unit applies to one of a first area, a second area, and a third area defined in a real space, or performs an area determination as to whether a displacement amount by which the at least one of the face orientation and the head position of the occupant detected by the feature detection unit has shifted from the at least one of the face orientation and the head position of the reference posture of the occupant applies to one of the first area, the second area, and the third area defined in the real space, a posture collapse determination unit that performs a posture collapse type determination, in which, in the case that the area determination unit determines,that the second range applies to the at least one of the face orientation and the head position of the occupant detected by the feature detection unit, or determines that the second range applies to the displacement amount, by a comparison between the at least one of the face orientation and the head position of the occupant detected by the feature detection unit and the at least one of the face orientation and the head position of the reference posture of the occupant, it is determined whether one of a plurality of posture collapse types applies to the posture of the occupant, a face change degree detection unit that detects a degree of change of the occupant's face based on the part of the face of the occupant detected by the feature detection unit, an error detection determination unit that determines, in the event that the posture collapse determination unit,that the posture of the occupant is one of the plurality of posture collapse types and the facial change degree detection unit has detected the change degree, determines that the judgment of the posture collapse determination unit is erroneous, and in the event that the posture collapse determination unit determines that the posture of the occupant is one of the plurality of posture collapse types and the facial change degree detection unit does not detect a change degree, determines that the determination of the posture collapse determination unit is not erroneous, and a problem posture judgment unit that, in the event that the area determination unit determines that the third area applies to at least one of the facial orientation and the head position of the occupant detected by the feature detection unit, the area determination unit determines that the third area applies to the displacement amount,or the error detection determination unit determines that the judgment of the posture collapse determination unit is not erroneous, determines that a problem posture of the occupant exists, and in the case where the area determination unit determines that the at least one of the facial orientation and the head position of the occupant detected by the feature detection unit applies to the first area, the area determination unit determines that the displacement amount applies to the first area, the posture collapse determination unit determines that none of the plurality of posture collapse types applies to the occupant's posture, or the error detection determination unit determines that the determination of the posture collapse determination unit is erroneous, determines that there is no problem posture of the occupant.

[0008] An occupant state determination system of the present disclosure includes a face detection unit that detects the face of the occupant in an image in which the face of an occupant of a vehicle is recorded, a feature detection unit that detects at least one of a face orientation and a head position of the occupant and facial parts based on the face of the occupant detected by the face detection unit in the image, a reference posture estimation unit that estimates at least one of the face orientation and the head position of the occupant, which are used as a reference posture, based on the at least one of the face orientation and the head position of the occupant detected by the feature detection unit within a predetermined period of time, an area determination unit that performs an area determination thereto,whether the at least one of the face orientation and the head position of the occupant detected by the feature detection unit applies to one of a first area, a second area, and a third area defined in a real space, or performs an area determination as to whether a displacement amount by which the at least one of the face orientation and the head position of the occupant detected by the feature detection unit has shifted from the at least one of the face orientation and the head position of the reference posture of the occupant applies to one of the first area, the second area, and the third area defined in the real space, a posture collapse determination unit that performs a posture collapse type determination, in which, in the case that the area determination unit determines,that the second range applies to the at least one of the face orientation and the head position of the occupant detected by the feature detection unit, or determines that the second range applies to the displacement amount, by a comparison between the at least one of the face orientation and the head position of the occupant detected by the feature detection unit and the at least one of the face orientation and the head position of the reference posture of the occupant, it is determined whether one of a plurality of posture collapse types applies to the posture of the occupant, a face change degree detection unit that detects a degree of change of the occupant's face based on the part of the face of the occupant detected by the feature detection unit, an error detection determination unit that determines, in the event that the posture collapse determination unit,that one of the plurality of posture collapse types applies to the posture of the occupant and the facial change degree detection unit has detected the degree of change, determines that the determination of the posture collapse determination unit is erroneous, and in the event that the posture collapse determination unit determines that one of the plurality of posture collapse types applies to the posture of the occupant and the facial change degree detection unit does not detect a degree of change, determines that the determination of the posture collapse determination unit is not erroneous, a problem posture determination unit that, in the event that the area determination unit determines that the third area applies to at least one of the facial orientation and the head position of the occupant detected by the feature detection unit, the area determination unit determines that the third area applies to the displacement amount,or the error detection determination unit determines that the determination of the posture collapse determination unit is not erroneous, determines that there is a problem posture of the occupant, and in the case where the area determination unit determines that the at least one of the facial orientation and the head position of the occupant detected by the feature detection unit applies to the first area, the area determination unit determines that the displacement amount applies to the first area, the posture collapse determination unit determines that none of the plurality of posture collapse types applies to the occupant's posture, or the error detection determination unit determines that the determination of the posture collapse determination unit is erroneous, determines that there is no problem posture of the occupant, and an output device installed in the vehicle and outputs a warning thereto,that the posture of the occupant is a problem posture, wherein the occupant condition determining device determines whether the posture of the occupant is a problem posture based on the image captured by the image capturing device, and in the event that it determines that the posture of the occupant is a problem posture, causes the output device to output the warning that the posture of the occupant is a problem posture.

[0009] An occupant state determination method of the present disclosure includes detecting a face of an occupant of a vehicle in an image in which the occupant's face is captured by a face detection unit, detecting at least one of a face orientation and a head position of the occupant and facial parts from the occupant's face detected by the face detection unit in the image, by a feature detection unit, estimating at least one of the face orientation and the head position of the occupant, which are used as a reference, as a reference posture from the at least one of the face orientation and the head position of the occupant detected by the feature detection unit within a predetermined period of time, by a reference posture estimation unit, performing an area determination therefor,whether the at least one of the face orientation and the head position of the occupant detected by the feature detection unit applies to one of a first area, a second area, and a third area defined in a real space, or a area determination as to whether a displacement amount by which the at least one of the face orientation and the head position of the occupant detected by the feature detection unit has shifted from the at least one of the face orientation and the head position of the reference posture of the occupant applies to one of the first area, the second area, and the third area defined in the real space, by a area determination unit, performing a posture collapse type determination in which, in the case where the area determination unit determines,that the second range applies to the at least one of the face orientation and the head position of the occupant detected by the feature detection unit, or determines that the second range applies to the displacement amount, by comparing the at least one of the face orientation and the head position of the occupant detected by the feature detection unit and the at least one of the face orientation and the head position of the reference posture of the occupant, it is determined whether one of a plurality of posture collapse types applies to the posture of the occupant, by a posture collapse determination unit, detecting a degree of change of the face of the occupant based on the part of the face of the occupant detected by the feature detection unit, by a face change degree detection unit, determining that the determination of the posture collapse determination unit is erroneous,If the posture collapse determination unit determines that one of the plurality of posture collapse types applies to the occupant's posture and the facial change degree detection unit has detected the degree of change, and that the determination of the posture collapse determination unit is not erroneous; If the posture collapse determination unit determines that one of the plurality of posture collapse types applies to the occupant's posture and the facial change degree detection unit does not detect a degree of change, by an error detection determination unit, and determining that a problem posture of the occupant exists; If the area determination unit determines that the third area applies to at least one of the facial orientation and the head position of the occupant detected by the feature detection unit, the area determination unit determines that the third area applies to the displacement amount;or the error detection determination unit determines that the determination of the posture collapse determination unit is not erroneous, and determines that there is no problem posture of the occupant if the area determination unit determines that the at least one of the facial orientation and the head position of the occupant detected by the feature detection unit falls within the first area, the area determination unit determines that the displacement amount falls within the first area, the posture collapse determination unit determines that none of the plurality of posture collapse types applies to the occupant's posture, or the error detection determination unit determines that the determination of the posture collapse determination unit is erroneous, by a problem posture determination unit.

[0010] A program of the present disclosure causes a computer to perform the following: detecting a face of an occupant of a vehicle in an image in which the occupant's face is captured by a face detection unit; detecting at least one of a face orientation and a head position of the occupant and facial parts from the occupant's face detected by the face detection unit in the image, by a feature detection unit; estimating at least one of the face orientation and the head position of the occupant, which are used as a reference, as a reference posture from the at least one of the face orientation and the head position of the occupant detected by the feature detection unit within a predetermined period of time, by a reference posture estimation unit; performing an area determination therefor,whether the at least one of the face orientation and the head position of the occupant detected by the feature detection unit applies to one of a first area, a second area, and a third area defined in a real space, or a area determination as to whether a displacement amount by which the at least one of the face orientation and the head position of the occupant detected by the feature detection unit has shifted from the at least one of the face orientation and the head position of the reference posture of the occupant applies to one of the first area, the second area, and the third area defined in the real space, by a area determination unit, performing a posture collapse type determination in which, in the case where the area determination unit determines,that the second range applies to the at least one of the face orientation and the head position of the occupant detected by the feature detection unit, or determines that the second range applies to the displacement amount, by comparing the at least one of the face orientation and the head position of the occupant detected by the feature detection unit and the at least one of the face orientation and the head position of the reference posture of the occupant, it is determined whether one of a plurality of posture collapse types applies to the posture of the occupant, by a posture collapse determination unit, detecting a degree of change of the face of the occupant based on the part of the face of the occupant detected by the feature detection unit, by a face change degree detection unit, determining that the determination of the posture collapse determination unit is erroneous,if the posture collapse determination unit determines that one of the plurality of posture collapse types applies to the posture of the occupant and the facial change degree detection unit has detected the degree of change, and determining that the determination of the posture collapse determination unit is not erroneous; if the posture collapse determination unit determines that one of the plurality of posture collapse types applies to the posture of the occupant and the facial change degree detection unit does not detect a degree of change, by an error detection determination unit, and determining that there is a posture error of the occupant; if the area determination unit determines that the third area applies to at least one of the facial orientation and the head position of the occupant detected by the feature detection unit, the area determination unit determines that the third area applies to the displacement amount;or the error detection determination unit determines that the determination of the posture collapse determination unit is not erroneous, and determines that there is no problem posture of the occupant if the area determination unit determines that the at least one of the facial orientation and the head position of the occupant detected by the feature detection unit falls within the first area, the area determination unit determines that the displacement amount falls within the first area, the posture collapse determination unit determines that none of the plurality of posture collapse types applies to the occupant's posture, or the error detection determination unit determines that the determination of the posture collapse determination unit is erroneous, by a problem posture determination unit. Effect of the invention

[0011] According to the occupant state determining device of the present disclosure, the detection accuracy of the occupant state can be increased. Short description of the characters

[0012] They show: Fig. 1 shows a configuration example of the occupant state determination device of a first embodiment; Fig. 2 is a view illustrating a first region, a second region, and a third region of a face alignment; Fig. 3 is a view illustrating a first region, a second region, and a third region of a head position; Fig. 4 is a flowchart illustrating the operation of the occupant state determining apparatus of the first embodiment; Fig. 5 an explanatory view of several postural collapse types; Fig. 6 shows an example of the hardware configuration of the occupant state determining device of the first embodiment; and Fig. 7 shows an example of the hardware configuration of the occupant state determination device of the first embodiment. Embodiment of the invention

[0013] With reference to the figures, a detailed description of an embodiment of the present disclosure now follows. First embodiment

[0014] The occupant state determining device of the first embodiment determines whether the posture of the occupant of the vehicle is a problem posture based on an image in which at least the face of an occupant of a vehicle is captured.

[0015] In the following embodiment, the occupant for whom the occupant state determining device determines whether there is a problem posture is, as an example, the driver of the vehicle.

[0016] Fig. 1 shows a configuration example of an occupant state determination device 1 of the first embodiment.

[0017] It is assumed that the occupant state determination device 1 of the first embodiment is installed in a vehicle.

[0018] The occupant state determination device 1 is connected to an image recording device 2 and an output device 3, wherein an occupant state determination system 100 is formed by the occupant state determination device 1, the image recording device 2 and the output device 3.

[0019] The image pickup device 2 is installed in a vehicle and arranged to capture at least an area where the driver's face should be located. In the first embodiment, it is assumed that the image pickup device 2 is installed, for example, in the region of a central portion in the width direction of a vehicle dashboard or in a center console. The image pickup device 2 can also simultaneously serve as a so-called DMS (driver monitoring system) installed for the purpose of monitoring the interior of the passenger compartment. The image pickup device 2 can be a visible light camera or an infrared camera. The image pickup device 2 outputs the captured images to the occupant state determination device 1.

[0020] The occupant state determination device 1 determines whether the driver's posture is a problem posture based on the image captured by the image pickup device 2. Details of the occupant state determination device 1 will be described later.

[0021] When the occupant state determining device 1 determines that the driver's posture is a problem posture, it causes the output device 3 to output a warning that the occupant's posture is a problem posture.

[0022] The output device 3 is installed in the vehicle and issues a warning that the posture of the occupant is a problem posture.

[0023] The output device 3 is a voice output device such as a loudspeaker or the like. The output device 3 may, for example, be part of an audio device provided in the vehicle. When information for issuing a warning (hereinafter referred to as "warning information") is output from the occupant condition determination device 1, the output device 3 outputs, for example, a warning sound or a voice message to notify that the driver's posture is a problem posture. The output device 3 may, for example, also be a display device such as a display or the like. When the warning information is output from the occupant condition determination device 1, the output device 3 displays, for example, a message to notify that the driver's posture is a problem posture.

[0024] The output device 3 may also be, for example, a direction indicator, a warning light, or a headlight provided on the vehicle exterior, or the like, visible to other vehicles. For example, the occupant condition determination device 1 may output the warning information to the output device 3 to notify persons outside the vehicle, such as occupants of other vehicles, that the driver of the vehicle is experiencing a problem condition.

[0025] The output device 3 may also be installed in other vehicles, for example. The occupant condition determination device 1 may transmit the warning information to the output devices 3 installed in other vehicles and perform a voice output or display, or the like, to notify the occupants of the other vehicles that the driver of the vehicle is experiencing a problem condition.

[0026] As in Fig. 1, the occupant state determination device 1 includes an image acquisition unit 11, a face detection unit 12, a feature detection unit 13, a reference posture estimation unit 14, a determination unit 15, and an output control unit 16.

[0027] The feature detection unit 13 includes a face orientation detection unit 131, a head position detection unit 132, a face position detection unit 133, and a detection reliability calculation unit 134.

[0028] The determination unit 15 includes an area determination unit 151, a posture collapse determination unit 152, a face change degree detection unit 153, an error detection determination unit 154, and a problem posture determination unit 155.

[0029] The image acquisition unit 11 acquires the image from the image pickup device 2.

[0030] The image acquisition unit 11 outputs the acquired image to the face detection unit 12.

[0031] The face detection unit 12 detects the driver's face in the image acquired by the image acquisition unit 11.

[0032] More specifically, the face detection unit 12 detects feature points of the driver's face in the image capture using known image recognition techniques such as edge detection or the like, which indicate facial parts of the driver. The facial parts include, among others, the outer corners of the eyes, the inner corners of the eyes, the nose, the mouth, the eyebrows, or the chin. The feature points of the driver's face are expressed, for example, by coordinates in the image capture. The face detection unit 12 can also detect a facial region of the driver. The facial region of the driver is, for example, a smallest rectangle that encloses the contour of the driver's face. The facial region of the driver is expressed, for example, by the coordinates of the four corners of the smallest rectangle in the image capture.

[0033] Since the installation position and angle of the image pickup device 2 are known in advance, the face detection unit 12 can distinguish the driver's face in the image pickup, even when multiple passengers are captured in the image pickup. The area in which the driver's face may be located in the image pickup (hereinafter referred to as the "driver detection area") is determined in advance, and the face detection unit 12 detects the driver's feature points and the driver's face area in the driver detection area using existing image recognition techniques.

[0034] The face detection unit 12 outputs the information about the driver's face (hereinafter referred to as "face information") to the feature detection unit 13. The face information is, for example, image recordings that have been provided with information from which the driver's feature points and facial area can be determined.

[0035] The feature detection unit 13 detects the features from which the posture of the occupant, here the driver, can be determined. The features for determining the driver's posture are a facial orientation of the driver, a head position of the driver, and facial parts of the driver. In the first embodiment, the facial orientation and head position of the driver detected by the feature detection unit 13 (hereinafter, the facial orientation of the driver detected by the feature detection unit 13 is referred to as the "detected facial orientation" and the head position of the driver detected by the feature detection unit 13 is referred to as the "detected head position") are, for example, the facial orientation and head position in real space. In the first embodiment, the position of the facial part of the driver detected by the feature detection unit 13 is, for example, the position of the facial part in the image recording.For example, in the first embodiment, the driver's face is the driver's eyes. The following description assumes that the driver's face is the driver's eyes.

[0036] The face orientation detection unit 131 detects the driver's face orientation in real space based on the facial information output from the face detection unit 12, specifically, the driver's face detected by the face detection unit 12 in the image capture. The face orientation detection unit 131 can detect the driver's face orientation using, for example, a known face orientation detection technique for detecting face orientation in image capture. The driver's face orientation is expressed, for example, by at least one of a yaw angle, which is a rotation angle with respect to an axis extending in the up-down direction of the vehicle, a pitch angle, which is a rotation angle about an axis extending in the left-right direction of the vehicle, and a roll angle, which is a rotation angle about a front-rear direction of the vehicle, determined in advance as a reference.The face orientation detection unit 131 outputs the detected face orientation information (hereinafter referred to as "face orientation information") to the reference posture estimation unit 14 and the determination unit 15. The face orientation information includes at least one of the yaw angle, the pitch angle, and the roll angle.

[0037] The head position detection unit 132 detects the driver's head position in real space based on the facial information output from the face detection unit 12, specifically, the driver's face detected by the face detection unit 12 in the captured image. In the first embodiment, the driver's head position in the captured image is indicated, for example, by the midpoint between the driver's eyebrows. The head position detection unit 132 detects, as the driver's head position, the point in real space corresponding to the midpoint between the driver's eyebrows in the captured image, for example. This is just one example, and the driver's head position in the captured image may also be indicated by the midpoint of the driver's facial area or the midpoint of a line connecting the two inner corners of the driver's eyes.In this case, the head position detecting unit 132 detects, as the driver's head position, for example, the point in real space corresponding to the center of the driver's facial area or the center of the line connecting the two inner corners of the driver's eyes.

[0038] The head position detection unit 132 can detect the driver's head position using a known coordinate conversion technique, for example, converting points in image captures into points in real space. The driver's head position is expressed, for example, by coordinates in real space. The head position detection unit 132 outputs information about the detected driver's head position (hereinafter referred to as "head position information") to the reference posture estimation unit 14 and the determination unit 15. The head position information includes the coordinate information of the driver's head position.

[0039] The face position detection unit 133 detects the position of the driver's eyes in the image based on the facial information output by the face detection unit 12, specifically, the driver's face detected by the face detection unit 12 in the image capture. The face detection unit 12 detects the position of the driver's right eye and the position of the driver's left eye in the image capture. In the first embodiment, the position of the driver's eyes (hereinafter, the position of the driver's eyes detected by the feature detection unit 13 is referred to as the "detected eye position") is the center point of the line in the image capture that connects the outer and inner corners of the driver's eyes. The face position detection unit 133 can detect the position of the driver's eyes using, for example, a known image recognition technique.The position of the driver's eyes is expressed, for example, by coordinates in the image capture. The batch position acquisition unit 133 outputs the acquired information on the position of the driver's eyes (hereinafter referred to as "batch position information") to the determination unit 15. The batch position information includes information on the coordinates of the position of the driver's left eye and right eye.

[0040] The detection reliability calculation unit 134 calculates a detection reliability indicating the reliability that the at least one of the face orientation, the head position, and the face part is the predetermined at least one of the face orientation, the head position, and the face part. The detection reliability calculation unit 134 outputs the calculated detection reliability to at least one of the reference posture estimation unit 14 and the determination unit 15. For calculating the detection reliability, the method specified in the following document, among others, can be used.In the document below, the detection reliability of the at least one of the face orientation, the head position, and the face part is calculated based on a degree of agreement between the at least one of the face orientation, the head position, and the face part learned in advance from large amounts of face images and the at least one of the face orientation, the head position, and the face part of the driver extracted from a shot of the driver.

[0041] “Rapid Object Detection using a Boosted Cascade of Simple Features,” 2001, Paul Viola et al.

[0042] As long as objects corresponding to facial parts can be detected from images in a method without the method mentioned in the above document and their detection reliability can be calculated, such a method can also be applied to the detection reliability calculation unit 134.

[0043] The reference posture estimation unit 14 estimates at least one of the driver's face orientation and head position detected within a predetermined period (hereinafter referred to as a "determined reference period") as a reference posture. The determined reference period is, for example, a time until a predetermined time (e.g., 10 seconds) elapses from the vehicle ignition switch being turned on. This is merely an example, and the determined reference period may also be a time (e.g., 3 seconds) for which, for example, the vehicle speed is at or above a predetermined speed (e.g., 25 km / h) and a steering angle is within a predetermined range (e.g., ±20 degrees).The at least one of the driver's face orientation and head position used as a reference is estimated as a reference posture based on the at least one of the detected face orientation and head position, but the reference posture may also be estimated based on at least one of the at least one of the detected face orientation and head position and the detection reliability of the face orientation and detection reliability of the head position.

[0044] For example, the reference posture estimation unit 14 estimates the average value of at least one of the detected face orientation and the detected head position in the specified reference period as the at least one of the detected face orientation and the detected head position at the reference posture of the driver (hereinafter, the face orientation detected at the reference posture of the driver is referred to as “reference face orientation” and the head position detected at the reference posture of the driver is referred to as “reference head position”).

[0045] The reference posture estimation unit 14 outputs information on the estimated reference posture of the driver (hereinafter referred to as "reference posture information") to the determination unit 15. The reference posture information includes information of at least one of the reference face orientation and the reference head position. The reference face orientation is expressed by at least one of the yaw angle, the pitch angle, and the roll angle. The reference head position is expressed, for example, by coordinates in real space. The reference posture information includes, as the driver's head position information, coordinate information of the driver's head position.

[0046] The area determination unit 151 performs area determination as to whether the at least one of the detected face orientation and the detected head position falls within one of a first area, a second area, and a third area set in the real space, or performs area determination as to whether a displacement amount by which the at least one of the detected face orientation and the detected head position has shifted from the at least one of the reference face orientation and the reference head position (hereinafter referred to as a displacement amount) falls within one of the first area, a second area, and a third area set in the real space.

[0047] The first range, the second range, and the third range are set for at least one of the yaw angle of the driver's face orientation, the pitch angle of the driver's face orientation, the roll angle of the driver's face orientation, a plane formed by a horizontal direction and a height direction of the driver's head position, a plane formed by the horizontal direction and a depth direction of the driver's head position, and a plane formed by the height direction and the depth direction of the driver's head position.

[0048] Fig. Figure 2 shows the first area, the second area, and the third area defined for the yaw angle of the driver's face orientation, the pitch angle of the driver's face orientation, and the roll angle of the driver's face orientation, respectively. The first area, the second area, and the third area indicate Fig. 2, a reference face orientation specified as 0° is set as the center point, and the first region, the second region, and the third region are sequentially defined from the reference face orientation in the order of increasing absolute value of the angle of the driver's face orientation. The second region has a larger absolute value of the angle from the reference face orientation than the first region, and the third region has a larger absolute value of the angle from the reference face orientation than the second region.

[0049] Fig. 3 shows the first area, the second area, and the third area, which are defined in the plane formed by the horizontal direction and the height direction of the driver's head position, the plane formed by the horizontal direction and the depth direction of the driver's head position, and the plane formed by the height direction and the depth direction of the driver's head position, respectively. The first area, the second area, and the third area have Fig. 3, a reference head position whose center is specified as the intersection point of the vertical axis and the horizontal axis, wherein the first range, the second range, and the third range are sequentially defined from the reference facial orientation in the order of increasing absolute value of the displacement amount of the driver's head position. The second range has a larger absolute value of the displacement amount from the reference head position than the first range, and the third range has a larger absolute value of the displacement amount from the reference head position than the second range. The horizontal direction, the height direction, and the depth direction for the driver's head position are assigned to the vertical axis and the horizontal axis, respectively.

[0050] The first range is set to a range within which at least one of the driver's face orientation and head position can be located when there is no malposture described below. The first range is set for at least one of the driver's face orientation yaw angle, the driver's face orientation pitch angle, the driver's face orientation roll angle, the plane formed by the horizontal direction and the height direction of the driver's head position, the plane formed by the horizontal direction and the depth direction of the driver's head position, and the plane formed by the height direction and the depth direction of the driver's head position, for example, 0 to 15° away from the reference face orientation and 0 to 50 mm away from the reference head position.

[0051] The second range is set to a range within which at least one of the facial orientation and head position of a driver without a bad posture and a driver with a bad posture can lie. The second range is set for at least one of the yaw angle of the driver's facial orientation, the pitch angle of the driver's facial orientation, the roll angle of the driver's facial orientation, the plane formed by the horizontal direction and the height direction of the driver's head position, the plane formed by the horizontal direction and the depth direction of the driver's head position, and the plane formed by the height direction and the depth direction of the driver's head position, for example, by 16 to 35° or 51 to 250 mm from the reference head position.

[0052] The third range is set to a range within which at least one of the driver's face orientation and head position may lie when a posture error occurs. The third range is set to at least one of the driver's face orientation yaw angle, the driver's face orientation pitch angle, the driver's face orientation roll angle, the plane formed by the horizontal direction and the height direction of the driver's head position, a plane formed by the horizontal direction and the depth direction of the driver's head position, and the plane formed by the height direction and the depth direction of the driver's head position, for example, 36 to 90° away from the reference face orientation and 251 to 350 mm away from the reference head position.

[0053] If at least one of the detected facial orientation and the detected head position falls within the first range or the third range, or if the displacement amount falls within the first range or the third range, the range determination unit 151 outputs the range determination result to the problem posture determination unit 155. If at least one of the detected facial orientation and the detected head position falls within the second range, or if the displacement amount falls within the second range, the range determination unit 151 outputs the range determination result to the posture collapse determination unit 152.The area determination result includes information that at least one of the detected face orientation and the detected head position is one of the first area, the second area, and the third area, or that the displacement amount is one of the first area, the second area, and the third area.

[0054] If the range determination unit 151 determines that at least one of the detected facial orientation and the detected head position falls within the second range, or the displacement amount falls within the second range, the posture breakdown determination unit 152 determines whether the driver's posture falls within one of a plurality of predetermined posture failure types (hereinafter referred to as "posture breakdown types") by comparing the at least one of the detected facial orientation and head position with at least one of the reference facial orientation and the reference head position. In the first embodiment, the determination by the posture breakdown determination unit 152 as to whether the driver's posture falls within one of a plurality of posture breakdown types is also referred to as "posture breakdown type determination."

[0055] In the multiple posture breakdown types, posture types are defined in advance, which are a problem posture resulting from the presence of a problem condition of the driver.

[0056] Fig. Figure 4 is an explanatory view of the multiple posture collapse types (partially amended version of “Doraibaa ijöjidökenchishisutemu kihonsekkeisho” (Document on the Basic Design of Automatic Driver Problem Detection Systems) of the Investigation Group for the Promotion of Modern Safety Vehicles of the Ministry of Transport, March 2018). As in Fig. As shown in Figure 4, there are many forms of poor posture. Fig. 4, a state in which the driver remains in a posture in which he falls forward and his face reaches the steering wheel is referred to as "falling on the face"; a state in which the driver remains in a posture in which his face is directed downwards is referred to as "head hanging posture"; a state in which the driver remains in a posture in which his upper body is tilted backwards and his face is directed upwards is referred to as "backbend posture"; a state in which the driver remains in a posture in which his upper body is curved upwards and his face is directed upwards is referred to as "curving posture"; a state in which the driver remains in a posture in which the driver's face is turned to the left or right is referred to as "sideways tilting of the neck"; a state in which the driver remains in a posture in which his upper body is turned to the left or right and his face is also turned in the same direction is referred to as "sideways tilting";and a condition in which the driver remains in a posture in which his upper body is slumped to the left or right is referred to as "lateral slumping". In the present disclosure, the conditions described in , Fig. 4, the postures "falling on the face," "head hanging posture," "back bending posture," "curving posture," "lateral neck tilting," "lateral tilting," and "lateral slumping" are used as the multiple posture collapse types determined in advance. The information on the multiple posture collapse types is stored in a location accessible by the posture collapse determination unit 152.

[0057] Specifically, the posture breakdown determination unit 152 determines which of the multiple posture breakdown types applies to the driver based on how much the detected facial orientation has changed from the reference facial orientation or how much the detected head position has changed from the reference head position. The conditions for determining which posture breakdown type applies when the detected facial orientation has changed from the reference facial orientation or how much the detected head position has changed from the reference head position are set in advance and stored in the posture breakdown determination unit 152. The posture breakdown determination unit 152 performs the posture breakdown type determination using, for example, a learning device such as an SVM (support vector machine) or the like.

[0058] The posture breakdown determination unit 152 outputs the posture breakdown type determination result to the fault detection determination unit 154 if one of the posture breakdown types applies to the driver's posture. The posture breakdown determination unit 152 outputs the posture breakdown type determination result to the problem posture determination unit 155 if one of the posture breakdown types applies to the driver's posture. The posture breakdown type determination result includes information about whether one of the incorrect posture types applies to the driver's posture.

[0059] The facial change degree detection unit 153 detects a change degree of the driver's face based on the driver's facial part detected by the feature detection unit 13.

[0060] A change in the driver's face means that the driver moves at least part of their face, so that the state of the driver's face changes, and the degree of change of the face indicates the extent to which the driver's face has changed within a predetermined period of time. For example, the change in the driver's face can be cited as the driver's blinking, and the degree of change of the driver's face can be cited as the number of times the driver blinks within the predetermined period of time. In the first embodiment, the change in the driver's face is cited as an example as the driver's blinking, and the degree of change of the driver's face is cited as the number of times the driver blinks within the predetermined period of time.

[0061] Specifically, the facial change degree detection unit 153 detects the degree of eye opening of the driver based on the detected eye position and determines whether the driver is blinking. The degree of eye opening can be detected, for example, as a value obtained by detecting the position of the inner corners of the eyes, the position of the outer corners of the eyes, and the position of the upper eyelids at the highest point; dividing a distance between a straight line connecting the inner corners of the eyes and the outer corners of the eyes and the position of the upper eyelids at the highest point by the distance between the inner corners of the eyes and the outer corners of the eyes to determine a degree of eye flattening; and dividing the degree of flattening by a predetermined value. The highest point of the upper eyelids is the point on the upper eyelids (vertex) that is farthest from the straight line connecting the inner corners of the eyes and the outer corners of the eyes.If the degree of eye opening is less than a predetermined threshold TH1, it is determined that the eyes are closed. If the degree of eye opening exceeds a predetermined threshold TH2, which is higher than the predetermined threshold TH1, it is determined that the eyes are open. If open eyes are detected after detecting closed eyes within a predetermined time, it is determined that blinking is occurring. If no open eyes are detected after detecting closed eyes within a predetermined time, it is determined that no blinking is occurring.

[0062] The degree of opening of the driver's eyes is detected based on the detected eye position and thus the presence of blinking of the driver can be detected, but the degree of opening of the driver's eyes and the presence of blinking of the driver can also be detected based on the detected eye position and the reliability of the detection of the facial area.

[0063] The face change degree detection unit 153 counts the number of times within a predetermined period of time that blinking is determined to occur (hereinafter referred to as blinking frequency), and if the blinking frequency is at or above a threshold, it is deemed to detect the change degree, while if the blinking frequency is below the threshold, it is deemed that no change degree of the face is detected.

[0064] The face change degree detecting unit 153 outputs the result of whether a change degree of the face has been detected (hereinafter referred to as “face change degree detection result”) to the error detection determining unit 154.

[0065] When the posture collapse determination unit 152 determines that the driver's posture is one of the plurality of posture collapse types and the facial change degree detection unit 153 detects a change degree, the error detection determination unit 154 determines that the driver's posture is not a posture collapse type, that is, the determination of the posture collapse determination unit 152 is erroneous. When the posture collapse determination unit 152 determines that the driver's posture is one of the plurality of posture collapse types and the facial change degree detection unit 153 detects no change degree, the error detection determination unit 154 determines that the driver's posture is a posture collapse type, that is, the determination of the posture collapse determination unit 152 is not erroneous.

[0066] The fault detection determination unit 154 outputs the determination result of whether the determination of the posture collapse determination unit 152 is faulty or not (hereinafter referred to as “detection fault determination result”) to the problem posture determination unit 155.

[0067] The problem posture determination unit 155 determines whether the driver's posture is a problem posture based on the region determination result outputted from the area determination unit 151, the posture breakdown type determination result outputted from the posture breakdown determination unit 152, and the detection error determination result outputted from the error detection determination unit 154.

[0068] The problem posture determination unit 155 determines that the driver has no problem posture when the area determination unit 151 determines that at least one of the driver's face orientation and head position detected by the feature detection unit 13 falls within the first area, the area determination unit 152 determines that the displacement amount falls within the first area, the posture breakdown determination unit 152 determines that the driver's posture does not fall within any of the plurality of posture breakdown types, or the error detection determination unit 154 determines that the determination of the posture breakdown determination unit 152 is erroneous.

[0069] The problem posture determining unit 155 determines, in a case where the area determining unit 151 determines that the at least one of the driver's face orientation and the head position detected by the feature detecting unit 13 falls within the third area, the area determining unit 151 determines that the displacement amount falls within the third area, or the error detecting determining unit 154 determines that the determination of the posture breakdown determining unit 152 is not erroneous, the time for which the state that the at least one of the driver's face orientation and the head position falls within the third area, the state that the displacement amount falls within the third area, or the state that the driver's posture is a faulty posture continues.For example, the problem posture determination unit 155 links the obtained area determination result or posture breakdown type determination result with information about the date and time of obtaining the area determination result or posture breakdown type determination result and stores it in a storage unit (not shown). The problem posture determination unit 155 can determine, from the stored area determination result or posture breakdown type determination result, the time for which the state that at least one of the driver's facial orientation and head position falls within the third area, the state that the displacement amount falls within the third area, or the state that the driver's posture is a malposture continues.When the time for which the state that at least one of the driver's face orientation and head position is in the third range, the state that the displacement amount is in the third range, or the state that the driver's posture is a malposture continues reaches a predetermined threshold (hereinafter referred to as “problem posture determination threshold”), the problem posture determination unit 155 determines that the driver has a problem posture.

[0070] The problem posture determination unit 155 outputs the determination result of whether the driver has a problem posture (hereinafter referred to as “problem posture determination result”) to the output control unit 16.

[0071] When the problem posture determining unit 155 determines that there is a problem posture of the driver, the output control unit 16 outputs warning information to the output device 3 to warn the driver.

[0072] The following is a description of the operation of the occupant state determining device 1 of the first embodiment. Fig. 5 is a flowchart for explaining the operation of the occupant state determining device 1 of the first embodiment.

[0073] The operation of the occupant state determination device 1 begins, for example, after the vehicle's ignition is turned on. Preferably, the occupant state determination processing is performed whenever an image is acquired from the image acquisition device 2, i.e., for each frame.

[0074] The image acquisition unit 11 acquires the image from the image pickup device 2 (step ST1).

[0075] The image acquisition unit 11 outputs the acquired image to the face detection unit 12.

[0076] In step ST1, the face detection unit 12 detects the driver's face in the image acquired by the image acquisition unit 11 (step ST2).

[0077] The face detection unit 12 outputs the face information to the feature detection unit 13.

[0078] In step ST2, the face orientation detection unit 131 detects the face orientation of the driver in the real space based on the face information output by the face detection unit 12, more specifically, the face of the driver detected by the face detection unit 12 in the image capture (step ST3).

[0079] The face orientation detection unit 131 outputs the face orientation information to the reference pose estimation unit 14 and the determination unit 15.

[0080] In step ST2, the head position detection unit 132 detects the head position of the driver in the real space based on the face information output by the face detection unit 12, more specifically the face of the driver detected by the face detection unit 12 in the image capture (step ST3).

[0081] The head position detection unit 132 outputs the head position information to the reference posture estimation unit 14 and the determination unit 15.

[0082] In step ST2, the head position detection unit 132 detects the head position of the driver in the real space based on the face information output by the face detection unit 12, more specifically the face of the driver detected by the face detection unit 12 in the image capture (step ST3).

[0083] The head position detection unit 132 outputs the head position information to the reference posture estimation unit 14 and the determination unit 15.

[0084] In step ST2, the part position detection unit 133 detects the position of the driver's face part in the real space based on the face information output by the face detection unit 12, more specifically the driver's face detected by the face detection unit 12 in the image capture (step ST3).

[0085] The batch position detection unit 133 outputs the batch position information to the determination unit 15.

[0086] The detection reliability calculation unit 134 calculates the detection reliability of at least one of the face orientation, the head position, and the face part based on the face information outputted from the face detection unit 12 in step ST2, or more specifically, based on the face of the driver in the image captured by the face detection unit 12 (step ST4).

[0087] The detection reliability calculation unit 134 outputs the detection reliability to at least one of the reference posture estimation unit 14 and the determination unit 15.

[0088] If the reference posture estimation unit 14 has not yet completed the estimation of the driver's reference posture ("NO" in step ST5), it estimates the driver's reference posture based on at least one of the driver's face orientation and the head position detected by the feature detection unit 13 within the specified reference period (as discussed above, hereinafter, the driver's face orientation detected by the feature detection unit 13 is referred to as "detected face orientation" and the driver's head position detected by the feature detection unit 13 is referred to as "detected head position") (step ST6).

[0089] As already discussed, the reference pose estimation unit 14 may also estimate the reference pose based on at least one of the at least one of the detected face orientation and head position and the detection reliability of the face orientation and the detection reliability of the head position.

[0090] Whether the reference posture estimation unit 14 has completed estimating the driver's reference posture can be determined, for example, based on a reference posture estimated flag. When the reference posture estimation unit 14 has estimated the driver's reference posture, the reference posture estimated flag can be set to "1," for example. The initial value of the reference posture estimated flag is "0," and the reference posture estimated flag is reset, for example, when the ignition of the occupant state determination device 1 is turned off. The reference posture estimated flag is stored in a location accessible to the occupant state determination device 1.

[0091] When the reference posture estimation unit 14 has completed estimating the reference posture of the driver (“YES” in step ST5), the area determination unit 151 performs the area determination as to whether at least one of the detected face orientation and the detected head position falls within one of the first area, the second area, and the third area set in the real space, or the area determination as to whether the displacement amount falls within one of the first area, the second area, and the third area set in the real space (step ST7).

[0092] When at least one of the detected face orientation and the detected head position is within the first range or the third range, or the displacement amount is within the first range or the third range, the range determination unit 151 outputs the range determination result to the problem pose determination unit 155.

[0093] When at least one of the detected face orientation and the detected head position is within the second range or the displacement amount is within the second range, the range determination unit 151 outputs the range determination result to the posture collapse determination unit 152.

[0094] When the range determining unit 151 determines that the at least one of the detected face orientation and the detected head position falls within the second range or the displacement amount falls within the second range (“second range” of step ST7), the posture collapse determining unit 152 determines whether the driver's posture falls within one of the plurality of preset types of malposture by comparing the at least one of the detected face orientation and the head position and the at least one of the reference face orientation and the reference head position.

[0095] The posture breakdown determination unit 152 outputs the posture breakdown type determination result to the failure detection determination unit 154 when one of the posture breakdown types applies to the driver's posture.

[0096] The posture collapse determination unit 152 outputs the posture collapse type determination result to the problem posture determination unit 155 in the case that one of the posture collapse types applies to the driver's posture.

[0097] When the driver's posture is one of the posture collapse types (“YES” in step ST8), the posture collapse determination unit 152, the face change degree detection unit 153 detects a change degree of the driver's face based on the driver's face part detected by the feature detection unit 13.

[0098] The face change degree detection unit 153 outputs the face change degree detection result to the error detection determination unit 154 (step ST9).

[0099] When the posture collapse determination unit 152 determines that the driver's posture is one of the plurality of posture collapse types and the face change degree detection unit 153 detects a change degree, the error detection determination unit 154 determines that the driver's posture is not a posture collapse type, and when the posture collapse determination unit 152 determines that the driver's posture is one of the plurality of posture collapse types and the face change degree detection unit 153 detects no change degree, it determines that the driver's posture is a posture collapse type.

[0100] The error detection determination unit 154 outputs the detection error determination result to the problem attitude determination unit 155 (step ST10).

[0101] The problem posture determination unit 155 determines that the driver has no problem posture in the case where the area determination unit 151 determines that at least one of the driver's facial orientation and head position detected by the feature detection unit 13 falls within the first area, or the area determination unit 151 determines that the displacement amount falls within the first area ("first area" in step ST7), the posture breakdown determination unit 152 determines that the driver's posture does not fall within any of the plurality of posture breakdown types ("NO" in step ST8), or the error detection determination unit 154 determines that the driver has no problem posture, that is, the determination of the posture breakdown determination unit 152 is erroneous ("NO" in step ST10).

[0102] The problem posture determination unit 155 determines, in a case where the area determination unit 151 determines that the at least one of the driver's face orientation and the head position detected by the feature detection unit 13 falls within the third area, the area determination unit 151 determines that the displacement amount falls within the third area (“third area” of step ST7), or the error detection determination unit 154 determines that the determination of the posture breakdown determination unit 152 is not erroneous (“YES” in step ST10), the time for which the state that the at least one of the driver's face orientation and the head position falls within the third area, or the state that the displacement amount falls within the third area, or the state that the driver's posture is a faulty posture continues (step ST11).

[0103] The problem posture determination unit 155 determines that there is a problem posture of the driver when the time for which the state that at least one of the driver's face orientation and head position is in the third range, or the state that the displacement amount is in the third range, or the state that the driver's posture is a bad posture continues reaches the problem posture determination threshold ("YES" in step ST11), and causes the output device 3 to output a warning that the driver's posture is a problem posture (step ST12).

[0104] The problem posture determination unit 155 determines that there is no problem posture of the driver in the case where the time for which the state that at least one of the driver's face orientation and head position is in the third range, the state that the displacement amount is in the third range, or the state that the driver's posture is a bad posture continues does not reach the problem posture determination threshold (“NO” in step ST11).

[0105] Fig. 6 and Fig. 7 shows an example of the hardware configuration of the occupant state determination device 1 of the first embodiment. In the first embodiment, the functions of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference posture estimation unit 14, the determination unit 15, and the output control unit 16 are implemented by a processing circuit 1001. Thus, the occupant state determination device 1 includes the processing circuit 1001 to perform control to determine, based on the image capture, whether the posture of the occupant is a problem posture. The processing circuit 1001 may be, as in Fig. 6 can be dedicated hardware, or as shown in Fig. 7 shows a processor 1004 that executes programs stored in a memory.

[0106] When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 is a single circuit, a plurality of circuits, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof.

[0107] When the processing circuit is the processor 1004, the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference pose estimation unit 14, the determination unit 15, and the output control unit 16 are implemented by software, firmware, or a combination of software and firmware. The software or firmware is recorded as a program and stored in the memory 1005. By reading and executing the programs stored in the memory 1005, the processor 1004 performs the functions of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference pose estimation unit 14, the determination unit 15, and the output control unit 16. The occupant state determination device 1 thus includes the memory 1005 in which the program is stored, by which, by execution by the processor 1004, the above-discussed steps ST1 to ST12 of Fig.6. Thus, it can be said that the programs stored in the memory 1005 cause the operations or methods of the processing of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference pose estimation unit 14, the determination unit 15, and the output control unit 16 to be executed. The memory 1005 is, for example, RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPPROM (Electrically Erasable Programmable Read Only Memory), or other non-volatile or volatile semiconductor memory, or a magnetic disk, a floppy disk, an optical disk, a mini disk, a DVD (Digital Versatile Disc), or the like.

[0108] The functions of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference pose estimation unit 14, the determination unit 15, and the output control unit 16 can also be implemented partly by dedicated hardware and partly by software or firmware. For example, it is also possible for the function of the image acquisition unit 11 and the output control unit 16 to be implemented with the processing circuit 1001 as dedicated hardware, and the functions of the face detection unit 12, the feature detection unit 13, the reference pose estimation unit 14, and the determination unit 15 to be implemented by the processor 1004 reading and executing programs from the memory 1005.

[0109] The storage unit not shown is formed, for example, by the memory 1005. The occupant state determination device 1 includes the image pickup device 2, the output device 3, or the like, and an input interface device 1002 and an output interface device 1003 for performing wired or wireless communication.

[0110] As described above, according to the first embodiment, there are the face detection unit 12, which detects the face of the occupant in an image in which the face of an occupant of a vehicle is recorded, the feature detection unit 13, which detects at least one of a face orientation and a head position of the occupant and facial parts based on the face of the occupant detected by the face detection unit 12 in the image recording, the reference posture estimation unit 14, which estimates at least one of the face orientation and the head position of the occupant based on the at least one of the face orientation and the head position of the occupant detected by the feature detection unit 13 within a predetermined period of time as a reference posture, the area determination unit 151, which performs an area determination therefor,whether the at least one of the face orientation and the head position of the occupant detected by the feature detection unit applies to one of a first area, a second area, and a third area defined in a real space, or performs an area determination as to whether a displacement amount by which the at least one of the face orientation and the head position of the occupant detected by the feature detection unit has shifted from the at least one of the face orientation and the head position of the reference posture of the occupant applies to one of the first area, the second area, and the third area defined in the real space, the posture collapse determination unit 152 which performs a posture collapse type determination in which, in the case where the area determination unit 151 determines,that the second range applies to at least one of the face orientation and the head position of the occupant detected by the feature detection unit 13, or determines that the second range applies to the displacement amount, by comparing the at least one of the face orientation and the head position of the occupant detected by the feature detection unit 13 and the at least one of the face orientation and the head position of the reference posture of the occupant, it is determined whether one of several posture collapse types applies to the posture of the occupant, the face change degree detection unit 153, which detects a degree of change of the occupant's face based on the part of the face of the occupant detected by the feature detection unit 13, the error detection determination unit 154, which determines in the event that the posture collapse determination unit 152,that one of the plurality of posture collapse types applies to the posture of the occupant and the facial change degree detection unit 153 has detected the degree of change, determines that the determination of the posture collapse determination unit 152 is erroneous, and in the event that the posture collapse determination unit 152 determines that one of the plurality of posture collapse types applies to the posture of the occupant and the facial change degree detection unit 153 does not detect a degree of change, determines that the determination of the posture collapse determination unit 152 is not erroneous, and the problem posture determination unit 155, which, in the event that the area determination unit 151 determines that the third area applies to at least one of the facial orientation and the head position of the occupant detected by the feature detection unit 13,that the third range applies to the displacement amount, or the error detection determination unit 154 determines that the determination of the posture collapse determination unit 152 is not erroneous, determines that a problem posture of the occupant exists, and in the case that the range determination unit 151 determines that the first range applies to at least one of the facial orientation and the head position of the occupant detected by the feature detection unit 13, the range determination unit 151 determines that the first range applies to the displacement amount, the posture collapse determination unit 152 determines that none of the plurality of posture collapse types applies to the occupant's posture, or the error detection determination unit 154 determines that the determination of the posture collapse determination unit 152 is erroneous, determines that there is no problem posture of the occupant.

[0111] The occupant state determining device 1 performs the posture collapse type determination of the posture collapse determining unit 152 and the detection error determination of the error detection determining unit 154 based on the at least one of the facial orientation and the head position of the driver detected by the feature detecting unit 13, or the posture collapse type determination of the posture collapse determining unit 152 and the detection error determination of the error detection determining unit 154 based on the displacement amount, not from all areas, but performs the area determination unit 151 as to whether the at least one of the facial orientation and the head position of the driver detected by the feature detecting unit 13 applies to one of the first area, the second area, and the third area, or the area determination as to whether the displacement amount applies to one of the first area,the second area and the third area, and changes the output destination of the scoping result depending on the scoping result.

[0112] Specifically, in the case where the area determining unit 151 determines that the second area applies to at least one of the driver's face orientation and head position detected by the feature detecting unit 13, or the second area applies to the displacement amount, it performs the posture collapse type determination of the posture collapse determining unit 152 and the detection error determination of the error detection determining unit 154 based on the at least one face orientation and head position detected by the feature detecting unit 13, or the posture collapse type determination of the posture collapse determining unit 152 and the detection error determination of the error detection determining unit 154 based on the displacement amount.

[0113] On the other hand, when the area determination unit 151 determines that the third area applies to at least one of the driver's face orientation and head position detected by the feature detection unit 13, or the third area applies to the displacement amount, it does not perform the posture breakdown type determination of the posture breakdown determination unit 152 and the detection error determination of the error detection determination unit 154, but outputs to the problem posture determination unit 155. When the time for which the state that the third area applies to at least one of the driver's face orientation and head position applies, or the state that the displacement amount applies to the third area, reaches the problem posture determination threshold, the problem posture determination unit 155 determines that the driver has a problem posture.

[0114] Therefore, the occupant state determining device 1 prevents erroneous detection of a posture problem in the second range, in which at least one of the facial orientation and the head position of both a driver with a posture problem and a driver without a posture problem can be found, in which it is determined that the driver's posture is a posture problem even though it is not a posture problem. In the third range, in which at least one of the facial orientation and the head position of the driver can be found when a posture problem occurs, it can be determined that the driver has a posture problem despite a change in the driver's face, that is, when the posture problem occurs during blinking or blinking continues after the posture problem occurs.Therefore, the failure to detect a posture error, which is not determined as a decrease in force despite the presence of the posture error, is prevented, thereby increasing the detection accuracy of the occupant condition.

[0115] In the first embodiment of the occupant state determination device 1, it is assumed that when the reference posture estimation unit 14 estimates the driver's reference posture, no correction of the reference posture is performed until the vehicle ignition is turned off. However, this is only an example, and the reference posture estimation unit 14 may also continuously monitor the reference posture and update the reference posture when the reference posture changes within a range that is not a driver misalignment.

[0116] In the above first embodiment, it is assumed that the reference posture estimation unit 14 estimates the driver's reference posture, but this is only an example. An administrator or the like may also set the reference posture in advance based on the face orientation, head position, and position of facial parts such as the eyes or the like of an ordinary driver, and store it in a location accessible to the determination unit 15.

[0117] In this case, a configuration is possible in which the occupant state determination device 1 does not include the reference posture estimation unit 14. However, with a configuration in which the reference posture estimation unit 14 estimates the reference posture, the occupant state determination device 1 can set the reference posture in a manner more adapted to the individual than in the configuration without the reference posture estimation unit 14. As a result, the occupant state determination device 1 can more accurately determine whether the driver's posture is problematic.

[0118] In the above first embodiment, the area determination unit 151 determines whether the at least one of the detected face orientation and the detected head position falls within one of the first area, the second area, and the third area set in the real space, or whether the displacement amount falls within one of the first area, the second area, and the third area set in the real space, but the area determination unit 151 may also determine the area determination by machine learning.

[0119] In the above embodiment, the occupant for whom the occupant state determination device 1 determines whether the occupant has a problem posture is the driver of the vehicle, but this is only an example. The occupant state determination device 1 may also determine whether the occupant has a problem posture for occupants of the vehicle other than the driver.

[0120] In the above first embodiment, the occupant state determination device 1 is an on-board device installed in the vehicle, and the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference posture estimation unit 14, the determination unit 15, and the output control unit 16 are part of the on-board device.

[0121] However, it is not limited to this, and a configuration is also possible in which some of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference pose estimation unit 14, the determination unit 15, and the output control unit 16 are installed in an on-board device of the vehicle and the rest are on a server connected to the on-board device via a network.

[0122] The image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference pose estimation unit 14, the determination unit 15, and the output control unit 16 may also all be present on one server. Explanation of reference symbols 1 occupant condition determination device 2 Image recording device 3 Dispensing device 5 Steering mechanism 6 Brake drive mechanism 11 Image acquisition unit 12 Face detection unit 13 Feature acquisition unit 131 Face orientation detection unit 132 Head position detection unit 133 Batch position recording unit 134 Detection reliability calculation unit 14 Reference attitude estimation unit 15 Determination unit 151 Area determination unit 152 Postural Collapse Determination Unit 153 Face change degree detection unit 154 Error detection determination unit 155 Problem-setting unit 16 Output control unit 100 Occupant Condition Determination System 1001 processing circuit 1002 Input interface device 1003 Output interface device 1004 processor 1005 memory QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2019-087018 A

[0004]

Claims

[1] An occupant condition determining device comprising: a face detection unit for detecting, in an image recording in which a face of an occupant of a vehicle is recorded, the face of the occupant; a feature detection unit for detecting, based on the face of the occupant detected by the face detection unit in the image recording, at least one of a face orientation and a head position of the occupant and a face part; a reference posture estimation unit for estimating, based on the at least one of the face orientation and the head position of the occupant detected within a predetermined period of time by the feature detection unit, at least one of the face orientation and the head position as a reference of the occupant as a reference posture; a range determination unit for performing a range determination as to whether the at least one of the face orientation and the head position of the occupant detected by the feature detection unit falls within one of a first range, a second range, and a third range set in a real space, or a range determination as to whether a displacement amount by which the at least one of the face orientation and the head position of the occupant detected by the feature detection unit has shifted from the at least one of the face orientation and the head position of the occupant in the reference posture falls within one of the first range, the second range, and the third range set in the real space; a posture collapse determination unit for performing posture collapse type determination, in which, in the case where the area determination unit determines that the second area applies to the at least one of the face orientation and the head position of the occupant detected by the feature detection unit, or determines that the second area applies to the displacement amount, it is determined whether the posture of the occupant is one of a plurality of posture collapse types by a comparison between the at least one of the face orientation and the head position of the occupant detected by the feature detection unit and the at least one of the face orientation and the head position of the occupant in the reference posture; a face change degree detection unit for detecting, based on the face part of the occupant detected by the feature detection unit, a change degree of the face of the occupant; an error detection determination unit for determining, when the posture collapse determination unit determines that one of the plurality of posture collapse types applies to the posture of the occupant and when the facial change degree detection unit has detected the change degree, that the determination of the posture collapse determination unit is erroneous, and when the posture collapse determination unit determines that one of the plurality of posture collapse types applies to the posture of the occupant and when the facial change degree detection unit does not detect the change degree, that the determination of the posture collapse determination unit is not erroneous; and a problem posture determining unit for determining, when the area determining unit determines that the third area applies to at least one of the facial orientation and the head position of the occupant detected by the feature detecting unit, if the area determining unit determines that the third area applies to the displacement amount, or if the error detection determining unit determines that the determination of the posture collapse determining unit is not erroneous, that the occupant is in a problem posture, and for determining, when the area determining unit determines that the first area applies to at least one of the facial orientation and the head position of the occupant detected by the feature detecting unit, if the area determining unit determines that the first area applies to the displacement amount, if the posture collapse determining unit determines,that none of the multiple posture collapse types applies to the occupant's posture, or if the error detection determination unit determines that the determination of the posture collapse determination unit is erroneous, that the occupant is not in the problem posture. [2] The occupant state determining device according to claim 1, wherein the facial orientation of the occupant is expressed by at least one of a yaw angle which is a rotation angle about an axis extending in the up-down direction of the vehicle, a pitch angle which is a rotation angle about an axis extending in the left-right direction of the vehicle, and a roll angle which is a rotation angle about a front-rear direction of the vehicle, which are references set in advance, and wherein the head position of the occupant is a position for which a point arbitrarily set on the face of the occupant in the image captured by the face detection unit has been converted into a point in the real space. [3] The occupant state determining device according to claim 2, wherein the first area, the second area, and the third area are set in at least one of the yaw angle of the occupant's face orientation, the pitch angle of the occupant's face orientation, the roll angle of the occupant's face orientation, a plane formed by a horizontal direction and a height direction of the occupant's head position, a plane formed by the horizontal direction and a depth direction of the occupant's head position, and a plane formed by the height direction and the depth direction of the occupant's head position, wherein a center point of each of the first region, the second region and the third region is the at least one of the face orientation and the head position of the occupant in the reference posture, and wherein the first area, the second area and the third area are successively determined from the at least one of the face orientation and the head position of the occupant in the reference posture. [4] The occupant state determining device according to claim 3, wherein in the second region, an absolute value of at least one of an angle and a displacement amount from the at least one of the face orientation and the head position of the occupant in the reference posture is larger than in the first region, and wherein in the third region, an absolute value of at least one of the angle and a displacement amount from the at least one of the face orientation and the head position of the occupant in the reference posture and the displacement amount is larger than in the second region. [5] The occupant state determining device according to any one of claims 1 to 4, wherein the degree of change of the face of the occupant is the number of times the occupant blinks within a predetermined period of time, wherein the face change degree detecting unit counts the number of times the occupant blinks within the predetermined period of time, and wherein, in the case that the number of times the occupant blinks is at or above a threshold value, the face change degree detecting unit is to detect the degree of change of the face, and in the case that the number of times the occupant blinks is below the threshold value, the face change degree detecting unit is not to detect the degree of change of the face. [6] The occupant state determining device according to any one of claims 1 to 5, wherein the area determining unit determines whether the at least one of the facial orientation and the head position of the occupant detected by the feature detecting unit applies to one of the first area, the second area, and the third area, or the displacement amount applies to one of the first area, the second area, and the third area, based on machine learning. [7] The occupant state determining device according to any one of claims 1 to 6, comprising an output control unit for outputting, when the trouble posture determining unit determines that the occupant is in the trouble posture, warning information to perform warning of the occupant. [8] An occupant state determining system comprising the occupant state determining device according to any one of claims 1 to 7, a pickup device installed in the vehicle so as to be able to detect at least the face of the occupant, and an output device installed in the vehicle for outputting a warning that the posture of the occupant is the problem posture, wherein the occupant state determining device determines whether the posture of the occupant is the problem posture based on the image captured by the pickup device, and when determining that the posture of the occupant is the problem posture, causes the output device to output the warning that the posture of the occupant is the problem posture. [9] An occupant condition determination method comprising the following steps: detecting, by a face detection unit, a face of an occupant of a vehicle in an image recording in which the face of the occupant is recorded; detecting, by a feature detection unit, at least one of a face orientation and a head position of the occupant and a face part using the face of the occupant detected by the face detection unit in the image recording; Estimating, by a reference posture estimation unit, at least one of the face orientation and the head position as a reference of the occupant, as a reference posture based on the at least one of the face orientation and the head position of the occupant detected within a predetermined period of time by the feature detection unit; Performing, by a range determining unit, a range determination as to whether the at least one of the face orientation and the head position of the occupant detected by the feature detecting unit falls within one of a first range, a second range, and a third range set in a real space, or a range determination as to whether a displacement amount by which the at least one of the face orientation and the head position of the occupant detected by the feature detecting unit has shifted from the at least one of the face orientation and the head position of the occupant in the reference posture falls within one of the first range, the second range, and the third range set in the real space; Performing, by a posture collapse determining unit, a posture collapse type determination, in which, when the area determining unit determines that the at least one of the face orientation and the head position of the occupant detected by the feature detecting unit applies to the second area, or determines that the displacement amount applies to the second area, it is determined by a comparison between the at least one of the face orientation and the head position of the occupant detected by the feature detecting unit and the at least one of the face orientation and the head position of the occupant in the reference posture, whether the posture of the occupant applies to one of a plurality of posture collapse types; detecting, by a face change degree detecting unit, a change degree of the occupant's face based on the part of the occupant's face detected by the feature detecting unit; Determining, by an error detection determination unit, that the determination of the posture collapse determination unit is erroneous if the posture collapse determination unit determines that one of the plurality of posture collapse types applies to the posture of the occupant and if the facial change degree detection unit has detected the change degree, and determining that the determination of the posture collapse determination unit is not erroneous if the posture collapse determination unit determines that one of the plurality of posture collapse types applies to the posture of the occupant and if the facial change degree detection unit does not detect the change degree; and Determining, by a problem posture determining unit, that the occupant is in a problem posture if the area determining unit determines that the third area applies to at least one of the facial orientation and the head position of the occupant detected by the feature detecting unit, if the area determining unit determines that the third area applies to the displacement amount, or if the error detection determining unit determines that the determination of the posture collapse determining unit is not erroneous, and determining that the occupant is not in the problem posture if the area determining unit determines that the first area applies to at least one of the facial orientation and the head position of the occupant detected by the feature detecting unit, if the area determining unit determines that the first area applies to the displacement amount,if the posture collapse determination unit determines that none of the plurality of posture collapse types applies to the posture of the occupant, or if the error detection determination unit determines that the determination of the posture collapse determination unit is erroneous. [10] A program that causes a computer to perform the following steps: detecting, by a face detection unit, a face of an occupant of a vehicle in an image recording in which the face of the occupant is recorded; detecting, by a feature detection unit, at least one of a face orientation and a head position of the occupant and a face part using the face of the occupant detected by the face detection unit in the image recording; Estimating, by a reference posture estimation unit, at least one of the face orientation and the head position as a reference of the occupant as a reference posture based on the at least one of the face orientation and the head position of the occupant detected by the feature detection unit within a predetermined period of time; Performing, by a range determining unit, a range determination as to whether the at least one of the face orientation and the head position of the occupant detected by the feature detecting unit falls within one of a first range, a second range, and a third range set in a real space, or a range determination as to whether a displacement amount by which the at least one of the face orientation and the head position of the occupant detected by the feature detecting unit has shifted from the at least one of the face orientation and the head position of the occupant in the reference posture falls within one of the first range, the second range, and the third range set in the real space; Performing, by a posture collapse determining unit, a posture collapse type determination, in which, when the area determining unit determines that the at least one of the face orientation and the head position of the occupant detected by the feature detecting unit applies to the second area, or determines that the displacement amount applies to the second area, it is determined whether one of a plurality of posture collapse types applies to the posture of the occupant by a comparison between the at least one of the face orientation and the head position of the occupant detected by the feature detecting unit and the at least one of the face orientation and the head position of the occupant in the reference posture; detecting, by a face change degree detecting unit, a change degree of the occupant's face based on the part of the occupant's face detected by the feature detecting unit; Determining, by an error detection determination unit, that the determination of the collapse determination unit is erroneous if the collapse determination unit determines that the posture of the occupant is one of the plurality of posture collapse types and if the facial change degree detection unit has detected the change degree, and determining that the determination of the posture collapse determination unit is not erroneous if the posture collapse determination unit determines that the posture of the occupant is one of the plurality of posture collapse types and if the facial change degree detection unit does not detect the change degree; and Determining, by a problem posture determining unit, that the occupant is in a problem posture if the area determining unit determines that the third area applies to at least one of the facial orientation and the head position of the occupant detected by the feature detecting unit, the area determining unit determines that the third area applies to the displacement amount, or if the error detection determining unit determines that the determination of the posture collapse determining unit is not erroneous, and determining that the occupant is not in a problem posture if the area determining unit determines that the first area applies to at least one of the facial orientation and the head position of the occupant detected by the feature detecting unit, if the area determining unit determines that the first area applies to the displacement amount,if the posture collapse determination unit determines that none of the plurality of posture collapse types applies to the posture of the occupant, or if the error detection determination unit determines that the determination of the posture collapse determination unit is erroneous.

Images