Driving assistance device and driving assistance method

Abstract

A driving assistance device (10) is provided with: a recognition unit (123) that recognizes the degree of likelihood that a driver (D) of a vehicle (S) has an article worn on the face on the basis of one or more feature points of the face extracted from one or more captured images generated by capturing a face image of the driver (D); a determination unit (124) that determines a state determination period for determining whether or not the state of the driver (D) is an abnormal state, and an abnormal range indicating the orientation of the face observed when the state of the driver (D) is an abnormal state, the state determination period and the abnormal range corresponding to the degree of likelihood; and a determination unit (125) that determines whether the state of the driver (D) is an abnormal state by determining whether the orientation of the face based on the one or more feature points is included within the abnormal range in the state determination period.

Description

Technical Field

[0001] The present invention relates to a driving assistance device and a driving assistance method for detecting an abnormal state of a driver of a moving body such as a vehicle. Background Art

[0002] Conventional driving assistance devices determine whether the state of the driver of a vehicle is an abnormal state by analyzing a captured image generated by capturing a face image of the driver and obtaining a change amount of the driver's posture or a closed-eye state (for example, JP-2021-157436-A). Summary of the Invention

[0003] Problems to be Solved by the Invention In the above driving assistance device, when the driver has a wearing article such as a mask on his or her face, the number of facial feature points of the driver detected in the captured image decreases because the face is blocked by the wearing article. Therefore, problems such as stopping the determination even when the driver's state is an abnormal state and mis-determining that the driver's state is an abnormal state even when the driver's state is not an abnormal state occur.

[0004] The present invention has been made in view of these problems, and an object thereof is to improve the determination accuracy of the state of a driver having a wearing article on his or her face.

[0005] Means for Solving the Problems A driving assistance device according to a first aspect of the present invention includes: an identification unit that identifies a likelihood degree that a driver of a vehicle has a wearing article on his or her face based on one or more feature points of the face extracted from one or more captured images generated by capturing a face image of the driver; a determination unit that determines a state determination period for determining whether the state of the driver is an abnormal state, and an abnormal range representing a facial orientation observed when the state of the driver is an abnormal state, the state determination period and the abnormal range corresponding to the likelihood degree; and a determination unit that determines whether the state of the driver is an abnormal state by determining whether a facial orientation based on one or more feature points is included in the abnormal range during the state determination period.

[0006] The identification unit may determine whether one or more mouth feature points representing the mouth of the face are included in the one or more extracted feature points at a predetermined time, and when it is determined that the mouth feature points are not included during a wearing determination period including a plurality of predetermined times, identify a first degree indicating that the driver has a wearing article on his or her face as the likelihood degree.

[0007] The recognition unit can determine whether one or more mouth feature points representing the mouth of the face are included in one or more extracted feature points at a predetermined time, and when making determinations including the determination of the presence of the mouth feature points and the determination of the absence of the mouth feature points during a wearing determination period including a plurality of predetermined times, recognize a second degree indicating the possibility that the driver has a wearing item on the face as the likelihood degree.

[0008] The recognition unit can determine whether the orientation of the face in the pitch direction with respect to the forward direction of the vehicle at a predetermined time is an upward orientation or a downward orientation, and when the determination of the upward orientation and the determination of the downward orientation alternate repeatedly during a wearing determination period including a plurality of predetermined times, recognize a second degree indicating the possibility that the driver has a wearing item on the face as the likelihood degree.

[0009] When the likelihood degree is the first degree, compared with the case where the likelihood degree is a degree different from the first degree, the determination unit can make the range of the orientation of the face in the pitch direction with respect to the forward direction of the vehicle smaller, and this range is represented by an abnormal range.

[0010] When the likelihood degree is the first degree, the determination unit can refrain from determining whether the driver's state is an abnormal state.

[0011] The determination unit can make the state determination period when the likelihood degree is the second degree longer than the case where the likelihood degree is a degree different from the second degree.

[0012] At each predetermined time in the state determination period, the determination unit can calculate the angular difference between: the angle of the orientation of the face in the pitch direction with respect to the forward direction of the vehicle at a time after a predetermined time interval; and the angle of the orientation of the face at a time before the predetermined time interval, and when the likelihood degree is the second degree and the angular difference is equal to or greater than the threshold value, make the state determination period longer.

[0013] At each predetermined time in the state determination period, the determination unit can calculate the angular difference between: the angle of the orientation of the face in the pitch direction with respect to the forward direction of the vehicle at a time after a predetermined time interval; and the angle of the orientation of the face at a time before the predetermined time interval, and when a plurality of angular differences calculated after the likelihood degree has changed from the second degree to an alternative degree during the state determination period are less than the threshold value, the determination unit can determine whether the driver's state is an abnormal state.

[0014] The determination unit can re-determine whether the driver's state is an abnormal state based on the determination result of whether the driver's line-of-sight orientation is the forward direction in the vehicle's forward direction, provided that the likelihood degree is the first degree indicating that the driver has a wearing item on the face, and it is determined that the driver's state is an abnormal state because the orientation of the face is included in the abnormal range.

[0015] The abnormal state can have multiple types of abnormal states, and for each type of abnormal state, the determination unit can determine the initial value of the state determination period such that the initial value of the abnormal state type increases as the time when the determination unit makes a determination increases, and determines the state determination period for each type of abnormal state by changing the initial value according to the likelihood degree.

[0016] At a predetermined moment, the recognition unit can determine: whether one or more mouth feature points representing the mouth of the face are included in the one or more extracted feature points; and whether the orientation of the face in the pitch direction relative to the vehicle's forward direction is an upward orientation or a downward orientation, and when the mouth feature points are included in each of the feature points extracted from each captured image and the orientation of the face in the pitch direction relative to the vehicle's forward direction does not repeatedly alternate between the upward orientation and the downward orientation during the wearing determination period including a plurality of predetermined moments, recognize the third degree indicating that the driver does not have a wearing item on the face as the likelihood degree.

[0017] At each predetermined moment included in the state determination period, the determination unit can determine whether the orientation of the face recognized by the recognition unit is included in the abnormal range determined by the determination unit, and when it is determined that the orientation of the face is included in the abnormal range at a predetermined ratio or more of the plurality of predetermined moments included in the state determination period, determine that the driver's state is an abnormal state.

[0018] When the likelihood degree is the second degree, the determination unit can determine whether the orientation of the face recognized by the recognition unit is included in the abnormal range corresponding to an alternative degree different from the second degree, where the second degree indicates the possibility that the driver has a wearing item on the face after the likelihood degree has changed from the second degree to the alternative degree.

[0019] The driving assistance method according to the second aspect of the present invention is a driving assistance method executed by a computer, the driving assistance method having: an identification step of identifying the likelihood degree that the driver of the vehicle has a wearing item on the face based on one or more feature points of the face extracted from one or more captured images generated by capturing the face image of the driver; a determination step of determining a state determination period for determining whether the state of the driver is an abnormal state, and an abnormal range indicating the orientation of the face observed when the state of the driver is an abnormal state, the state determination period and the abnormal range corresponding to the likelihood degree; and a determination step of determining whether the state of the driver is an abnormal state by determining whether the orientation of the face based on one or more feature points includes the abnormal range during the state determination period.

[0020] Effects of the Invention The effect achieved by the present invention is that the determination accuracy of the state of the driver wearing an item on the face is improved. Description of the Drawings

[0021] Figure 1 It is a diagram for explaining the overview of the vehicle S according to the present embodiment.

[0022] Figure 2A 、 Figure 2B and Figure 2C They are diagrams for explaining the orientation of the face of the driver D, respectively.

[0023] Figure 3 It is a diagram showing an example of a driver wearing a mask M as a wearing item on the face.

[0024] Figure 4 It is a diagram for explaining the operation of identifying the likelihood degree.

[0025] Figure 5 It is a diagram for explaining the operation of determining the state determination period.

[0026] Figure 6 It is a sequence diagram showing the operations executed by the identification unit 123 for identifying the likelihood degree.

[0027] Figure 7 It is a sequence diagram showing the operations executed by the determination unit 124 for determining the abnormal range and the state determination period.

[0028] Figure 8 It is a diagram showing the initial range of the state determination period according to the modified example.

[0029] Description of Reference Numerals 1 Image capturing device 2 Notification device 3 Input device 4 Vehicle control device 10 Driving assistance device 11 Storage unit 12 Control unit 121 Acquisition unit 122 Extraction unit 123 Identification unit 124 Judgment unit 125 Determination unit 126 Instruction unit Detailed implementation manner

[0030] <Overview of vehicle S> Figure 1 It is a diagram for explaining the overview of vehicle S according to this embodiment. Figure 1 The vehicle S shown in the figure includes an image capturing device 1, a notification device 2, an input device 3, a vehicle control device 4, and a driving assistance device 10. The vehicle S has a function of determining whether the state of the driver of the vehicle S is an abnormal state. The abnormal state is a state where the driver's posture is deformed and the driver is unable to drive the vehicle S. For example, the abnormal state is any one of a forward fall posture, a looking-down posture, a backward tilt posture, a shrimp-shaped posterior arch posture, a lateral tilt posture, and a lateral fall posture of the driver.

[0031] The image capturing device 1 is a camera that can capture an image of an object both day and night, and is arranged in front of the driver's seat in the forward direction of the vehicle S and is set lower than the face position of the driver on the driver's seat in the height direction of the vehicle S. For example, the image capturing device 1 is an infrared camera. The image capturing device 1 generates a captured image of the face of the driver sitting on the driver's seat at a predetermined control moment (for example, after every 0.1 second), and outputs the captured image to the driving assistance device 10.

[0032] The notification device 2 is a device for notifying the driver that the driver is in an abnormal state. For example, when obtaining state information indicating that the driver's state is an abnormal state from the driving assistance device 10, the notification device 2 causes the speaker of the notification device 2 to output a warning sound and causes the display of the notification device 2 to display a warning message, etc. For example, when obtaining operation information indicating that the input device 3 has accepted an operation for stopping the output of the warning sound and the display of the warning message from the driving assistance device 10, the notification device 2 stops the output of the warning sound and the display of the warning message.

[0033] The input device 3 is a device for accepting an operation performed by the driver, and includes, for example, an operation button or a touch panel provided to at least any one of a steering device of the vehicle S and an instrument panel of the vehicle S. For example, in the case of accepting an operation of stopping outputting a warning sound and displaying a warning message from a driver who recognizes that his / her state is an abnormal state, the input device 3 outputs operation information indicating the operation to the driving assistance device 10.

[0034] For example, the vehicle control device 4 is a device including one or more processors, such as a CPU (Central Processing Unit) or an ECU (Electronic Control Unit). For example, the vehicle control device 4 accelerates / decelerates the vehicle S according to an operation performed by the driver. For example, in the case where instruction information indicating an instruction for stopping the vehicle S is acquired from the driving assistance device 10, the vehicle control device 4 stops the vehicle S.

[0035] The driving assistance device 10 performs a process of making an abnormal state determination based on a captured image generated by capturing a facial image of the driver and acquired from the image capturing device 1. For example, in the case where it is determined that the state of the driver is an abnormal state, the driving assistance device 10 causes the notification device 2 to start outputting a warning sound and displaying a warning message, causes the vehicle control device 4 to stop the vehicle S, etc. The driving assistance device 10 may have a housing including electronic components, or may be a printed circuit board on which the electronic components are implemented.

[0036] For example, the driving assistance device 10 identifies the orientation of the face based on the positions of the eyes, mouth, nose, and facial contour (facial lines) of the driver extracted from the captured image, and determines whether the orientation of the face is within a predetermined range, thereby determining whether the state of the driver is an abnormal state. For example, the orientation of the face is represented by the orientation in the pitch direction, the yaw direction, and the roll direction relative to the forward direction, and is represented by the angle in each direction relative to the reference direction.

[0037] Figure 2A , Figure 2B and Figure 2C Each of them is a diagram for explaining the orientation of the driver D's face. Figure 2A shows the orientation of the face in the pitch direction relative to the advancing direction of the vehicle S, Figure 2B shows the orientation of the face in the yaw direction relative to the advancing direction of the vehicle S, Figure 2C The orientation of the face in the roll direction relative to the advancing direction of the vehicle S is shown. Figure 2C It is a view from the driver from the front side to the rear side of the forward direction of the vehicle S moving forward. Figure 2A , Figure 2B and Figure 2CAs shown, the angles in the pitch direction, yaw direction, and roll direction with respect to the forward direction of the vehicle S are expressed as angles measured with respect to a reference direction (zero degree), which is consistent with the direction of the face of the driver D when looking forward in the forward direction. Note that in the following description, "the pitch direction, yaw direction, and roll direction with respect to the forward direction of the vehicle S" is sometimes abbreviated as "the pitch direction, yaw direction, and roll direction".

[0038] Meanwhile, in some cases, the driver D has a wearing item on his or her face, such as a mask or headphones. Figure 3 is a diagram showing an example of a driver with a wearing item mask M on the face. As in Figure 3 shown, in the case where the driver D has the mask M, even if the driving assistance device 10 can correctly identify the orientation of the face in the yaw direction and roll direction based on the eye position of the driver D, there is a possibility that the driving assistance device 10 misidentifies the orientation of the face in the pitch direction by misidentifying the irregularity of the mask M as the facial contour.

[0039] In addition, in some cases, the driver D wears the mask M and either the nose or the mouth is not covered, and also has a wearing item different from the mask M (such as headphones) while wearing the mask M. In this case, the driving assistance device 10 correctly identifies or misidentifies the orientation of the face in the pitch direction according to the orientation of the face of the driver D, so it takes a long time to correctly identify the orientation of the face in the pitch direction. Therefore, there is a possibility that even when the driver D is driving in a normal posture, the driving assistance device 10 misdetermines that the state of the driver D is an abnormal state.

[0040] In view of this, the driving assistance device 10 identifies the likelihood degree that the driver D has a wearing item on his or her face. In the following description, "the likelihood degree that the driver has a wearing item on his or her face" is sometimes abbreviated as "the likelihood degree". The driving assistance device 10 identifies the first degree, the second degree, or the third degree as the likelihood degree. For example, the first degree represents a state where it is estimated that the possibility that the driver D has a wearing item on his or her face is 100%. For example, the second degree represents a state where it is estimated that there is a possibility that the driver D has a wearing item on his or her face (that is, the possibility that the driver has a wearing item on his or her face is greater than 0 and less than 100%). The third degree represents a state where it is estimated that the possibility that the driver D has a wearing item on his or her face is 0%.

[0041] Then, based on the likelihood degree, the driving assistance device 10 determines a state determination period for determining an abnormal state and an abnormal range for determining the orientation of the face of the driver in an abnormal state. For example, when the likelihood degree is at the first level, the driving assistance device 10 makes the abnormal range smaller than when the likelihood degree is at an alternative level. Since the driving assistance device 10 operates in this way, even if the irregularity of the worn item is misrecognized as the facial contour, the driving assistance device 10 can reduce the mis-determination of the state of the driver D because the abnormal range is smaller.

[0042] In addition, for example, when the likelihood degree is at the second level, the driving assistance device 10 makes the state determination period longer than when the likelihood degree is at an alternative level. Since the driving assistance device 10 operates in this way, even when it takes a long time to correctly recognize the facial orientation based on the state of the driver D having a worn item, the driving assistance device 10 can correctly determine the state of the driver D. Therefore, the driving assistance device 10 can improve the determination accuracy regarding whether the state of the driver D is an abnormal state.

[0043] The configuration of the driving assistance device 10 and the operations it performs will be described in detail below.

[0044] <Configuration of the driving assistance device 10> As Figure 1 shown, the driving assistance device 10 has a storage unit 11 and a control unit 12. The control unit 12 has an acquisition unit 121, an extraction unit 122, an identification unit 123, a determination unit 124, a determination unit 125, and an instruction unit 126.

[0045] The storage unit 11 has a storage medium, such as a ROM (Read-Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive). Programs to be executed by the control unit 12 are stored in the storage unit 11. Various types of information for making an abnormal state determination are stored in the storage unit 11.

[0046] For example, the control unit 12 is a processor, such as a CPU or an ECU. By executing the programs stored in the storage unit 11, the control unit 12 functions as the acquisition unit 121, the extraction unit 122, the identification unit 123, the determination unit 124, the determination unit 125, and the instruction unit 126. Note that the control unit 12 can be configured using one processor, or a combination of multiple processors or one or more processors and electronic circuits.

[0047] The configuration of each part implemented by the control unit 12 will be described below.

[0048] The acquisition unit 121 acquires one or more captured images generated by the image capturing device 1 by capturing the face image of the driver D of the vehicle S. For example, the acquisition unit 121 acquires one or more captured images by acquiring the captured images at a predetermined control time. For example, the predetermined control time occurs every 0.1 seconds. The extraction unit 122 extracts one or more feature points of the face (hereinafter referred to as "facial feature points") from the one or more captured images acquired by the acquisition unit 121. For example, the facial feature points include the positions of the components of the face (such as eyebrows, eyes, nose, and mouth), the position of the facial contour, and the orientation of the line of sight. For example, the extraction unit 122 extracts one or more facial feature points at a predetermined control time.

[0049] The recognition unit 123 recognizes the position and orientation of the face of the driver D based on the one or more facial feature points extracted from the captured image by the extraction unit 122. For example, the recognition unit 123 recognizes the position of the face of the driver D based on the position of the facial contour included in the facial feature points. For example, the recognition unit 123 recognizes the orientation of the face in the pitch direction with respect to the forward direction of the vehicle S based on the position of the facial contour and the position of the mouth included in the facial feature points. For example, the recognition unit 123 recognizes the orientation of the face in the yaw direction and the roll direction with respect to the forward direction of the vehicle S based on the position of the eyes included in the facial feature points.

[0050] The recognition unit 123 recognizes the likelihood degree that the driver D has a wearing item on his face based on the one or more facial feature points extracted from the captured image by the extraction unit 122. For example, at a predetermined control time, the recognition unit 123 determines whether one or more mouth feature points representing the mouth of the face are included in the one or more facial feature points extracted by the extraction unit 122. Then, for example, in the case where it is determined that the mouth feature points are not included in the wearing determination period including a plurality of predetermined control times, the recognition unit 123 recognizes the first degree indicating that the driver D has a wearing item on his face as the likelihood degree. For example, in the case where the control time occurs every 0.1 seconds, the duration of the wearing determination period is equal to or longer than 0.2 seconds and equal to or shorter than 0.5 seconds.

[0051] Figure 4 is a diagram for explaining the operation of recognizing the likelihood degree. Figure 4 The horizontal axis of represents time, Figure 4 The vertical axis of represents "mouth feature points" and "orientation of the face (pitch direction)", "mouth feature points" is the result of determining whether the mouth feature points are included in the facial feature points, and "orientation of the face (pitch direction)" is the result of recognizing the orientation of the face in the pitch direction with respect to the forward direction of the vehicle S. Figure 4The "orientation of the face (pitch direction)" shown in [Figure] indicates a positive value when the orientation of the face of the driver D is upward, and a negative value when the orientation of the face of the driver D is downward. For example, the recognition unit 123 determines that the mouth feature points are not included in the face feature points extracted from the plurality of captured images by the extraction unit 122 during the wearing determination period P11 from time T11 to time T12, and recognizes the first degree as the likelihood degree.

[0052] For example, in the case where it is determined that one or more mouth feature points are included in one or more face feature points and it is determined that the mouth feature points are not included in the face feature points during the wearing determination period, the recognition unit 123 recognizes the second degree indicating the possibility that the driver D has a wearable on his / her face as the likelihood degree. The state of "the possibility that the driver D has a wearable on his / her face" is a state where the driver D may or may not have a wearable on his / her face. For example, such a state includes: the case where the driver D has a wearable on his / her face but the nose or mouth is not covered, the case where the driver D has a plurality of wearables on his / her face, and the case where it is easy to misidentify the wearable as a constituent element of the face (such as the nose) due to the similarity between the shape of the wearable and the shape of the irregular part of the face. For example, in Figure 4 During the wearing determination period P12 from time T12 to time T14 shown in [Figure], the recognition unit 123 determines that the mouth feature points are not included at a time before time T13 and determines that the mouth feature points are included at a time after time T13. Then, the recognition unit 123 recognizes the second degree as the likelihood degree.

[0053] The recognition unit 123 may recognize the second degree as the likelihood degree based on the orientation of the face in the pitch direction with respect to the forward direction of the vehicle S. For example, at a predetermined control time, the recognition unit 123 determines whether the orientation of the face in the pitch direction with respect to the forward direction of the vehicle S is upward or downward. Then, for example, during the wearing determination period, in the case where it is repeatedly and alternately determined that the orientation of the face is upward and determined that the orientation of the face is downward, the recognition unit 123 recognizes the second degree as the likelihood degree.

[0054] For example, the recognition unit 123 recognizes that in Figure 4In the wearing determination period P13 from time T14 to time T15 shown in the figure, the orientation of the face in the pitch direction with respect to the forward direction of the vehicle S repeatedly alternates between an upward orientation and a downward orientation (so-called wobbling occurs). Then, for example, even if it is determined that the mouth feature points are included in each captured image during the wearing determination period P13, the recognition unit 123 will recognize the second degree as the likelihood degree due to the occurrence of wobbling. Due to the operation performed by the recognition unit 123 in this way, even if the extraction unit 122 misextracts the facial feature points, the recognition unit 123 can estimate whether the facial feature points are misextracted based on the orientation of the face, and thus can reduce the misrecognition of the likelihood degree.

[0055] For example, at a predetermined moment, the recognition unit 123 determines whether one or more mouth feature points representing the mouth of the face are included in the one or more extracted facial feature points, and whether the orientation of the face in the pitch direction with respect to the forward direction of the vehicle S is an upward orientation or a downward orientation. Then, for example, the recognition unit 123 recognizes the third degree as the likelihood degree in the following case, that is, during the wearing determination period including a plurality of predetermined moments, the mouth feature points are included in each facial feature point extracted from each captured image, and the orientation of the face in the pitch direction with respect to the forward direction of the vehicle S does not repeatedly alternate between an upward orientation and a downward orientation. The third degree represents the degree to which the driver D does not have a wearing article on his or her face.

[0056] The determination unit 124 determines an abnormal range corresponding to the likelihood degree recognized by the recognition unit 123, and this abnormal range represents the orientation of the face observed when the state of the driver D is an abnormal state. The initial range in each of the pitch direction, yaw direction, and roll direction with respect to the forward direction of the vehicle S is specified in the abnormal range and stored in the storage unit 11. For example, when the likelihood degree is the second degree or the third degree, the determination unit 124 determines the initial range of the abnormal range stored in the storage unit 11 as the abnormal range.

[0057] For example, when the likelihood degree is the first degree, compared with the case where the likelihood degree is a degree different from the first degree (i.e., the second degree or the third degree), the determination unit 124 makes the range of the orientation of the face in the pitch direction with respect to the forward direction of the vehicle S smaller, and this range is represented by the abnormal range. For example, when the likelihood degree is the first degree, the determination unit 124 identifies the range of the orientation of the face included in the initial range of the abnormal range in the pitch direction with respect to the forward direction of the vehicle S, and determines a range smaller than this range as the abnormal range in the pitch direction. Specifically, when the initial range of the abnormal range in the pitch direction with respect to the forward direction of the vehicle S is "equal to or greater than -90 degrees and less than -20 degrees", since the likelihood degree is the first degree, the determination unit 124 determines "equal to or greater than -90 degrees and less than -30 degrees" as the abnormal range in the pitch direction.

[0058] Due to the operation performed by the determination unit 124 in this way, the determination unit 125 can make the range used to determine that the state of the driver D is an abnormal state smaller, and thus can reduce the mis-determination of the state of the driver D based on the orientation of the face mis-identified by the identification unit 123. Note that when the likelihood degree is the first degree, the determination unit 124 determines the initial range stored in the storage unit 11 as the abnormal range of the orientation of the face in the yaw direction and the roll direction with respect to the forward direction of the vehicle S.

[0059] The determination unit 124 determines a state determination period corresponding to the likelihood degree identified by the identification unit 123, and this state determination period is used to determine whether the state of the driver D is an abnormal state. For example, the initial value of the duration of the state determination period is equal to or longer than 3 seconds and equal to or shorter than 7 seconds, and is stored in the storage unit 11. For example, when the likelihood degree is the first degree or the third degree, the determination unit 124 determines the initial value of the state determination period stored in the storage unit 11 as the state determination period.

[0060] For example, in the case where the likelihood degree is the second degree, the determination unit 124 makes the state determination period longer than the case where the likelihood degree is a degree different from the second degree (i.e., the first degree or the third degree). For example, in the case where the likelihood degree is the second degree, the determination unit 124 determines the period obtained by multiplying the initial value of the state determination period stored in the storage unit 11 by 2 as the state determination period. Due to the operation performed by the determination unit 124 in this way, the determination unit 124 can increase the possibility that the likelihood degree switches from the second degree to an alternative degree during the state determination period. Therefore, the determination unit 125 can increase the possibility that abnormal state determination can be performed after the state switches from the state where the recognition unit 123 may misrecognize the orientation of the face to the state where the recognition unit 123 can easily and correctly recognize the orientation of the face, and thus can reduce the mis-determination of the state of the driver D.

[0061] The determination unit 124 can determine the state determination period based on the angle of the orientation of the face recognized by the recognition unit 123 and the likelihood degree. For example, at a predetermined control time in the state determination period, the determination unit 124 calculates the angle difference between: the angle of the orientation of the face in the pitch direction with respect to the forward direction of the vehicle S at a time after a predetermined control time interval, and the angle of the orientation of the face at a time before the predetermined control time interval. Then, for example, the determination unit 124 makes the state determination period longer when the likelihood degree recognized by the recognition unit 123 is the second degree and the calculated angle difference is equal to or greater than a threshold value. The threshold value is the minimum value of the angle difference, based on which it can be estimated that the recognition unit 123 misrecognizes the orientation of the face due to the wearing item on the face of the driver D, and this value is determined by experiments or simulations.

[0062] Figure 5 is a diagram for explaining the operation of determining the state determination period. Figure 5 The horizontal axis of represents time. Figure 5 The vertical axis of represents: the "likelihood degree" recognized by the recognition unit 123 for each wearing determination period; the "orientation of the face (pitch direction)" representing the orientation angle of the face in the pitch direction with respect to the forward direction of the vehicle S; and "abnormal determination", which are the times when the determination unit 125 has performed abnormal state determination. Figure 5 The "×" shown in represents the time when the abnormal state determination has been cancelled, Figure 5 The "○" shown in represents the time when the abnormal state determination has been performed.

[0063] For example, the determination unit 124 identifies that the angular difference between the orientation angle D2 of the face at time T23 and the orientation angle D1 of the face at time T22, which is a predetermined control time interval before time T23, is equal to or greater than a threshold value, and the likelihood degree from time T22 to time T23 is the second degree. Then, at time T23, the determination unit 124 makes the state determination period longer by changing the state determination period from period P21 to period P22. Similarly, the determination unit 124 identifies that the angular difference between the orientation angle D4 of the face at time T26 and the orientation angle D3 of the face at time T25 is equal to or greater than the threshold value, and the likelihood degree from time T25 to time T26 is the second degree. Then, at time T26, the determination unit 124 further changes the state determination period from period P22 to period P23.

[0064] Next, since the calculated angular difference from time T27 to time T28 is less than the threshold value, the determination unit 124 does not change the state determination period starting from period P23, but determines a new state determination period at time T28. Due to the operations performed by the determination unit 124 as described above, the determination unit 124 can extend the period including the time when the recognition unit 123 may misrecognize the face orientation of the driver D. Therefore, the determination unit 125 can perform the abnormal state determination at a time when the recognition unit 123 can easily and correctly recognize the face orientation during the state determination period determined by the determination unit 124, and thus can improve the accuracy of the abnormal state determination. Note that the differences between the period P21, the period P22 determined by the determination unit 124, and the period P21, as well as the difference between the period P23 and the period P22, can be the same or different.

[0065] The determination unit 124 can make the state determination period longer when the calculated angular difference is equal to or greater than the threshold value and the likelihood degree is the first degree. Due to the operations performed by the determination unit 124 in this way, the determination unit 124 can make the time for determining whether the state of the driver D is an abnormal state longer when the driver D has a wearing item on his / her face. Therefore, the possibility that the determination unit 125 performs the abnormal state determination at a time when the recognition unit 123 has misrecognized the face orientation can be reduced, and thus the accuracy of the abnormal state determination can be improved.

[0066] The determination unit 125 determines whether the state of the driver D is an abnormal state by determining whether the orientation of the face based on the facial feature points during the state determination period is included within an abnormal range. "The orientation of the face is included within an abnormal range" means that the orientation of the face with respect to at least one of the orientation of the face in the pitch direction relative to the forward direction, the orientation of the face in the yaw direction relative to the forward direction, and the orientation of the face in the roll direction relative to the forward direction is included within the abnormal range corresponding to the orientation of the face.

[0067] For example, at each predetermined control moment included in the state determination period, the determination unit 125 determines whether the orientation of the face recognized by the recognition unit 123 is included within the abnormal range determined by the determination unit 124. Then, for example, among the multiple control moments included in the state determination period, when it is determined that the orientation of the face is included within the abnormal range at a predetermined ratio or more of the control moments, the determination unit 125 determines that the state of the driver D is an abnormal state. On the other hand, for example, among the multiple control moments included in the state determination period, when it is determined that the orientation of the face is included within the abnormal range at a ratio lower than the predetermined ratio of the control moments, the determination unit 125 determines that the state of the driver D is not an abnormal state. The predetermined ratio is a ratio determined through experiments or simulations.

[0068] For example, when the likelihood degree recognized by the recognition unit 123 is the first degree or the third degree, the determination unit 125 determines whether the orientation of the face recognized by the recognition unit 123 is included within the abnormal range determined by the determination unit 124 according to the likelihood degree. For example, when the likelihood degree is the second degree, after the likelihood degree has changed from the second degree to an alternative degree different from the second degree, the determination unit 125 determines whether the orientation of the face recognized by the recognition unit 123 is included within the abnormal range corresponding to the alternative degree. Due to the operations performed by the determination unit 125 in this way, in a state where the determination of whether the driver D has a wearing item on his face is unclear, the determination unit 125 can reduce the execution of the abnormal state determination.

[0069] When the angular difference between multiple orientations of the face calculated after the likelihood degree has changed from the second degree to an alternative degree during the state determination period is less than the threshold value, the determination unit 125 can determine whether the state of the driver D is an abnormal state. For example, the determination unit 124 calculates the Figure 5After the degree has changed from the second degree to the first degree during the time from time T27 to time T28 in the state determination period P23 shown, the angular difference in the orientation of the face in the pitch direction with respect to the forward direction of the vehicle S. Then, since the angular difference in the orientation of the face calculated at or after time T27 is less than the threshold value in the state determination period P23, the determination unit 125 determines whether the orientation of the face recognized by the recognition unit 123 at or after time T27 is included in the abnormal range corresponding to the first degree determined by the determination unit 124.

[0070] Due to the operation performed by the determination unit 125 as described above, the determination unit 125 can reduce the execution of the abnormal state determination in a state where it is unclear whether the driver D has a wearing item on his / her face. In addition, the determination unit 125 can reduce the execution of the abnormal state determination in a state where the recognition unit 123 has misrecognized the orientation of the face of the driver D. Therefore, the determination unit 125 can improve the accuracy of the abnormal state determination.

[0071] At the same time, even if the recognition unit 123 misrecognizes the posture of the driver D (i.e., the position or orientation of the face) because the driver D has a wearing item on his / her face and the mouth feature points are not included in the face feature points, the orientation of the line of sight of the driver D included in the face feature points can be correctly recognized in some cases. In view of this, after performing the abnormal state determination based on the orientation of the face of the driver D, the determination unit 125 can re-determine whether the state of the driver D is an abnormal state based on the orientation of the line of sight of the driver D.

[0072] For example, the determination unit 125 re-determines whether the state of the driver D is an abnormal state on the premise that the likelihood degree is the first degree and it is determined that the state of the driver D is an abnormal state because the orientation of the face is included in the abnormal range. For example, based on the determination result of whether the orientation of the line of sight of the driver D included in the face feature points is the forward direction in the forward direction of the vehicle S, the determination unit 125 re-determines whether the state of the driver D is an abnormal state. For example, when the orientation of the line of sight of the driver D is the forward direction in the forward direction of the vehicle S, the determination unit 125 re-determines that the state of the driver D is not an abnormal state, and when the orientation of the line of sight of the driver D is a direction different from the forward direction in the forward direction of the vehicle S, the determination unit 125 re-determines that the state of the driver D is an abnormal state.

[0073] For example, in a case where the result of determining the abnormal state before re-determination is different from the result of re-determination, the determination unit 125 regards the result of re-determination as the determination result. Since the determination unit 125 operates in this way, even when the mouth feature points are not included in the facial feature points, the determination unit 125 can highly accurately determine whether the state of the driver D is an abnormal state. In addition, in a case where the likelihood degree is the first degree, the determination unit 125 may not determine whether the state of the driver D is an abnormal state. Since the determination unit 125 operates in this way, the determination unit 125 can reduce the possibility of mis-determining the state of the driver D.

[0074] Based on the determination result determined by the determination unit 125, the instruction unit 126 notifies the driver D that the state of the driver D is an abnormal state and stops the vehicle S, etc. For example, in a case where the determination unit 125 determines that the state of the driver D is an abnormal state, the instruction unit 126 causes the speaker of the notification device 2 to output a warning sound by outputting state information indicating that the state of the driver D is an abnormal state to the notification device 2. For example, the instruction unit 126 causes the display of the notification device 2 to display a warning message by outputting state information to the notification device 2.

[0075] In a case where operation information indicating that the driver D has performed an operation to stop the warning sound and the warning message is acquired from the input device 3 within a predetermined time from the time when the state information is output, the instruction unit 126 causes the notification device 2 to stop outputting the warning sound and displaying the warning message. In a case where the operation information is not acquired within the predetermined time, the instruction unit 126 stops the vehicle S by outputting instruction information indicating an instruction for stopping the vehicle S to the vehicle control device 4. The predetermined time is a time (for example, 10 seconds) that is predicted to be the maximum time value for the driver D to perform an operation after receiving the warning sound and the warning message, and is stored in the storage unit 11.

[0076] <Process sequence executed on the driving assistance device 10> Figure 6 and Figure 7 is a diagram of an example of a process sequence executed by the driving assistance device 10. Figure 6 The process sequence shown in Figure 7 is a sequence representing the operation of identifying the likelihood degree executed by the identification unit 123, and

[0077] First, using Figure 6An operation of the recognition likelihood degree performed by the recognition unit 123 will be described. The extraction unit 122 extracts facial feature points from the image acquired by the acquisition unit 121 (S11). The recognition unit 123 recognizes the mouth feature points included in the facial feature points extracted by the extraction unit 122 (S12). The recognition unit 123 recognizes the orientation of the face based on the facial feature points (S13). When the wearing determination period has not elapsed (No in S14), the acquisition unit 121, the extraction unit 122, and the recognition unit 123 repeat the operations of steps S11 to S13.

[0078] When the wearing determination period has elapsed (Yes in S14), the recognition unit 123 determines whether the mouth feature points are included in each of the facial feature points extracted by the extraction unit 122 during the wearing determination period. When the mouth feature points are not included in each of the facial feature points extracted during the wearing determination period (Yes in S15), that is, when the mouth feature points are not included in all of the facial feature points extracted during the wearing determination period, the recognition unit 123 recognizes the first degree indicating that the driver D has a wearing item on his face as the likelihood degree (S16). When the mouth feature points are not one or more of the facial feature points not included in the facial feature points extracted during the wearing determination period (No in S15), the recognition unit 123 determines whether the mouth feature points are included in each facial feature point (that is, whether the mouth feature points are included in all of the facial feature points extracted during the wearing determination period).

[0079] When the mouth feature points are included in each of the facial feature points extracted during the wearing determination period (Yes in S17), that is, when one or more mouth feature points are included in all of the facial feature points extracted during the wearing determination period, the recognition unit 123 recognizes the third degree indicating that the driver D does not have a wearing item on his face as the likelihood degree (S18). When the mouth feature points are not one or more of the facial feature points not included in the facial feature points extracted during the wearing determination period in S17 (No in S17), that is, when the facial feature points including the mouth feature points and the facial feature points not including the mouth feature points are included in the facial feature points extracted during the wearing determination period, the recognition unit 123 recognizes the second degree indicating the possibility that the driver D has a wearing item on his face as the likelihood degree (S19).

[0080] Next, use Figure 7The operations of the determination range of determination anomalies and the state determination period performed by the determination unit 124 will be described. By referring to the initial range of the anomaly range and the initial value of the state determination period stored in the storage unit 11, the determination unit 124 determines the initial range as the anomaly range and the initial value as the state determination period (S21). When the likelihood degree recognized by the recognition unit 123 is the first degree (Yes in S22), the determination unit 124 changes the anomaly range by making the range of the orientation of the face in the pitch direction with respect to the forward direction of the vehicle S smaller, which is represented by the anomaly range (S23). When the likelihood degree recognized by the recognition unit 123 is not the first degree (No in S22), the determination unit 124 determines whether the likelihood degree is the second degree (S24).

[0081] When the likelihood degree is neither the first degree nor the second degree (No in S24), that is, when the likelihood degree is the third degree, the determination unit 124 ends the process. When the likelihood degree is the second degree (Yes in S24), the determination unit 124 calculates the angle difference in the state determination period based on the angle of the orientation of the face recognized by the recognition unit 123 (S25). The angle difference is the difference between the angle of the orientation of the face in the pitch direction with respect to the forward direction of the vehicle S at a time after a predetermined control time interval and the angle of the orientation of the face at a time before the predetermined control time interval. When the angle difference calculated by the determination unit 124 is equal to or greater than the threshold value (Yes in S26), the determination unit 124 extends the state determination period (S27). When the angle difference calculated by the determination unit 124 is less than the threshold value (No in S26), the determination unit 124 ends the process.

[0082] <Variant example> Although the operations of determining a state determination period to determine whether the state of the driver D is an abnormal state have been described above, this is not the only example. When there are multiple types of abnormal states, the determination unit 124 can determine the state determination period for each type of abnormal state.

[0083] Figure 8 It is a diagram showing the initial value of the state determination period according to the variant example. Figure 8 It shows the types of abnormal states and the initial ranges of the state determination periods corresponding to each type of abnormal state. As Figure 8As shown, when the abnormal state type is the "forward fall posture", the orientation of the face and the orientation of the line of sight are likely to be the same orientation, so the state determination period can be shortened. On the other hand, when the abnormal state type is the "looking-down posture", the orientation of the face and the orientation of the line of sight become different orientations in some cases. Therefore, the time for re-determination increases, and the state determination period becomes longer. Note that Figure 8 The initial range of the state determination period shown in Figure 8 can be stored in the storage unit 11.

[0084] For example, the determination unit 124 determines the initial value (initial range) of the state determination period for each type of abnormal state. For example, the determination unit 124 determines the initial value (initial range) of the state determination period such that the initial value of the abnormal state increases as the time for the determination unit 125 to make a determination increases. Then, for example, the determination unit 124 determines the state determination period for each type of abnormal state by changing the initial value (initial range) according to the degree of likelihood.

[0085] Specifically, by referring to the storage unit 11, the determination unit 124 determines the first state determination period of "3 seconds" corresponding to the "forward fall posture" and the second state determination period of "7 seconds" corresponding to the "looking-down posture". Then, for example, when the degree of likelihood is the second degree, the determination unit 124 determines the "6 seconds" obtained by multiplying the first state determination period by 2 as the new first state determination period, and determines the "14 seconds" obtained by multiplying the second state determination period by 2 as the new second state determination period. Due to the operations performed by the determination unit 124 in this way, the determination unit 124 can determine the state determination period suitable for the abnormal state type, and thus can further accurately determine whether the state of the driver D is an abnormal state.

[0086] <Effects achieved by the driving assistance device 10> As described above, the driving assistance device 10 includes: an identification unit 123 that identifies the degree of likelihood that the driver D of the vehicle S wears an item (mask) on the face based on one or more feature points of the face extracted from one or more captured images generated by capturing the face image of the driver D; a determination unit 124 that determines the state determination period for determining whether the state of the driver D is an abnormal state, and an abnormal range representing the face orientation observed when the state of the driver D is an abnormal state, the state determination period and the abnormal range corresponding to the degree of likelihood; and a determination unit 125 that determines whether the state of the driver D is an abnormal state by determining whether the orientation of the face based on one or more feature points is included in the abnormal range during the determined state determination period.

[0087] By being configured in this way, the driving assistance device 10 can determine an abnormal range and a state determination period corresponding to the likelihood degree of the driver D having a worn item. Then, by making the abnormal range smaller, the driving assistance device 10 can reduce the possibility of mis-determining the state of the driver D due to the small abnormal range, even if the irregularity of the worn item is mis-recognized as a facial contour. Further, by extending the state determination period, the driving assistance device 10 can determine the state of the driver D after the state changes from a state of mis-recognizing the orientation of the face to a state where the face can be correctly recognized. Therefore, the driving assistance device 10 can highly accurately determine whether the state of the driver D is an abnormal state.

[0088] Although the present invention has been described using embodiments so far, the technical scope of the present invention is not limited to the description scope of the above embodiments, but various modifications and changes can be made within the scope of the gist. For example, all or part of the devices can be configured in any unit in a functionally or physically distributed / integrated manner. Further, embodiments of the present invention also include new embodiments generated by any combination of a plurality of embodiments. The effects of the new embodiments generated by the combination combine the effects of the original embodiments.

Claims

1. A driving assistance device, comprising: an identification unit that identifies a likelihood degree that the driver has a wearing item on the face based on one or more feature points of the face extracted from one or more captured images generated by capturing a face image of the driver of the vehicle; a determination unit that determines a state determination period for determining whether the state of the driver is an abnormal state and an abnormal range representing an orientation of the face observed when the state of the driver is an abnormal state, the state determination period and the abnormal range corresponding to the likelihood degree; and a determination unit that determines whether the state of the driver is an abnormal state by determining whether an orientation of the face based on the one or more feature points is included in the abnormal range during the state determination period.

2. The driving assistance device according to claim 1, wherein, The identification unit determines whether one or more mouth feature points representing the mouth of the face are included in the one or more extracted feature points at a predetermined time, and when it is determined that the mouth feature points are not included during a wearing determination period including a plurality of predetermined times, identifies a first degree indicating that the driver has a wearing item on the face as the likelihood degree.

3. The driving assistance device according to claim 1, wherein, The identification unit determines whether one or more mouth feature points representing the mouth of the face are included in the one or more extracted feature points at a predetermined time, and when both a determination including the mouth feature points and a determination not including the mouth feature points are made during a wearing determination period including a plurality of predetermined times, identifies a second degree indicating a possibility that the driver has a wearing item on the face as the likelihood degree.

4. The driving assistance device according to claim 1, wherein, The identification unit determines whether the orientation of the face in the pitch direction with respect to the forward direction of the vehicle at a predetermined time is an upward orientation or a downward orientation, and when the determination of the upward orientation and the determination of the downward orientation of the orientation repeatedly alternate during a wearing determination period including a plurality of predetermined times, identifies a second degree indicating a possibility that the driver has a wearing item on the face as the likelihood degree.

5. The driving assistance device according to claim 2, wherein, When the likelihood degree is the first degree, the determination unit makes a smaller range of the orientation of the face in the pitch direction with respect to the forward direction of the vehicle, the range being represented by the abnormal range, compared to the case where the likelihood degree is a degree different from the first degree.

6. The driving assistance device according to claim 2, wherein, When the likelihood degree is the first degree, the determination unit does not determine whether the state of the driver is an abnormal state.

7. The driving assistance device according to claim 3 or 4, wherein, The determination unit makes the state determination period when the likelihood degree is the second degree longer than the case where the likelihood degree is a degree different from the second degree.

8. The driving assistance device according to claim 3 or 4, wherein, At each predetermined time during the state determination period, the determination unit calculates an angular difference between: an angle of the orientation of the face in the pitch direction with respect to the forward direction of the vehicle at a time after a predetermined time interval, and an angle of the orientation of the face at a time before the predetermined time interval; When the likelihood level is the second level and the angular difference is equal to or greater than the threshold, the state determination period is made longer.

9. The driving assistance device according to claim 3 or 4, wherein At each predetermined moment in the state determination period, the determination unit calculates the angular difference between: the angle of the orientation of the face in the pitch direction with respect to the forward direction of the vehicle at a time after a predetermined time interval, and the angle of the orientation of the face at a time before the predetermined time interval, and The determination unit determines whether the state of the driver is an abnormal state when a plurality of angular differences calculated after the likelihood level has changed from the second level to an alternative level during the state determination period are less than the threshold.

10. The driving assistance device according to claim 1, wherein, The determination unit re-determines whether the state of the driver is an abnormal state based on the determination result of whether the orientation of the driver's line of sight is the forward direction in the forward direction of the vehicle, provided that the likelihood level is the first level indicating that the driver has a wearing item on the face, and it is determined that the state of the driver is an abnormal state because the orientation of the face is included in the abnormal range.

11. The driving assistance device according to claim 1, wherein The abnormal state has multiple types of abnormal states, and For each type of abnormal state, the determination unit determines the initial value of the state determination period such that the initial value for the type of abnormal state increases as the time when the determination unit makes a determination increases, and determines the state determination period for each type of abnormal state by changing the initial value according to the likelihood level.

12. The driving assistance device according to claim 1, wherein, At a predetermined moment, the recognition unit determines whether one or more mouth feature points representing the mouth of the face are included in one or more extracted feature points; And whether the orientation of the face in the pitch direction with respect to the forward direction of the vehicle is an upward orientation or a downward orientation, and when the mouth feature points are included in each feature point extracted from each captured image and the orientation of the face in the pitch direction with respect to the forward direction of the vehicle does not repeatedly alternate between an upward orientation and a downward orientation during the wearing determination period including a plurality of predetermined moments, the third level indicating that the driver does not have a wearing item on the face is recognized as the likelihood level.

13. The driving assistance device according to claim 1, wherein, At each predetermined moment included in the state determination period, the determination unit determines whether the orientation of the face recognized by the recognition unit is included in the abnormal range determined by the determination unit, and when it is determined that the orientation of the face is included in the abnormal range at a predetermined ratio or more of the plurality of predetermined moments included in the state determination period, it is determined that the state of the driver is an abnormal state.

14. The driving assistance device according to claim 1, wherein, In a case where the likelihood degree is a second degree, the determination unit determines whether the orientation of the face recognized by the recognition unit is included in an abnormal range corresponding to an alternative degree different from the second degree, and the second degree indicates the possibility that the driver has a wearing item on the face after the likelihood degree has changed from the second degree to the alternative degree.

15. A driving assistance method executed by a computer, the driving assistance method including: a recognition step of recognizing a likelihood degree that the driver has a wearing item on the face based on one or more feature points of the face extracted from one or more captured images generated by capturing a face image of the driver of the vehicle; a determination step of determining a state determination period for determining whether the state of the driver is an abnormal state and an abnormal range indicating the orientation of the face observed in a case where the state of the driver is an abnormal state, the state determination period and the abnormal range corresponding to the likelihood degree; and a determination step of determining whether the state of the driver is an abnormal state by determining whether the orientation of the face based on the one or more feature points is included in the abnormal range during the state determination period.

Images