Driving support device and driving support method

Abstract

To improve the accuracy of determining the status of drivers who wear items on their face.SOLUTION: A driving support device 10 includes: a specification unit 123 that specifies a degree of probability that a driver D wears a facial wear on his / her face based on facial feature points extracted from a captured image generated by capturing the face of the driver D of a vehicle S; a decision unit 124 that decides, in accordance with the degree of probability, a state determination period for determining whether or not the state of the driver D is an abnormal state, and an abnormal range indicating an orientation of the face of the driver D when the state of the driver D is an abnormal state; and a determination unit 125 that determines whether or not the state of the driver D is the abnormal state by determining, in the state determination period, whether or not the orientation of the face based on the feature points is included in the abnormal range.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a driving support device and a driving support method for detecting an abnormal state of a driver of a moving body such as a vehicle.

Background Art

[0002] The driving support device of Patent Document 1 analyzes a captured image generated by capturing the face of a vehicle driver to obtain the amount of change in the driver's posture and the closed-eye state, thereby determining whether the driver's state is an abnormal state.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above driving support device, when the driver wears an item such as a mask on the face, the face is covered by the item, so the number of facial feature points detected from the captured image decreases, resulting in problems such as aborting the determination even if the driver's state is an abnormal state or misjudging the driver's state as an abnormal state.

[0005] Therefore, the present invention has been made in view of these points, and an object thereof is to improve the accuracy of determining the state of a driver wearing an item on the face.

Means for Solving the Problems

[0006] The driving support device according to the first aspect of the present invention includes a specifying unit that specifies the degree of probability that the driver is wearing an item on the face based on the feature points of the face extracted from the captured image generated by capturing the face of the driver of the vehicle, a state determination period that determines whether the state of the driver is an abnormal state according to the degree of probability, and an abnormal range that indicates the orientation of the face when the state of the driver is an abnormal state, and a determination unit that determines whether the state of the driver is an abnormal state by determining whether the orientation of the face based on the feature points is included in the abnormal range during the state determination period.

[0007] The specifying unit determines, at a predetermined cycle, whether the extracted feature points include a mouth feature point indicating the mouth of the face, and if it is determined that the mouth feature point is not included in a wearing determination period including a plurality of the predetermined cycles, the specifying unit may specify a first degree indicating that the driver is wearing an item on the face as the degree of probability.

[0008] The specifying unit determines, at a predetermined cycle, whether the extracted feature points include a mouth feature point indicating the mouth of the face, and if it is determined that the mouth feature point is included and then determined that the mouth feature point is not included in a wearing determination period including a plurality of the predetermined cycles, the specifying unit may specify a second degree indicating that there is a possibility that the driver is wearing an item on the face as the degree of probability.

[0009] The specifying unit determines, at a predetermined cycle, whether the orientation of the face in the pitch direction with respect to the traveling direction of the vehicle is upward or downward, and if the upward determination and the downward determination are alternately repeated in a wearing determination period including a plurality of the predetermined cycles, the specifying unit may specify a second degree indicating that there is a possibility that the driver is wearing an item on the face as the degree of probability.

[0010] When the degree of probability is the first degree, the determination unit may reduce the range of the orientation of the face in the pitch direction with respect to the traveling direction of the vehicle indicated by the abnormal range, compared to the case where the degree of probability is a degree different from the first degree.

[0011] When the degree of probability is the first degree, the determination unit may not determine whether the state of the driver is an abnormal state.

[0012] When the degree of probability is the second degree, the determination unit may extend the state determination period, compared to the case where the degree of probability is a degree different from the second degree.

[0013] The determination unit calculates, for each cycle during the state determination period, the angular difference between the angle of the orientation of the face in the pitch direction with respect to the traveling direction of the vehicle at the time when the cycle has elapsed and the angle of the orientation of the face at the time one cycle before that time, and when the degree of probability is the second degree and the angular difference is equal to or greater than a threshold value, the determination unit may extend the state determination period.

[0014] The determination unit calculates, for each cycle during the state determination period, the angular difference between the angle of the orientation of the face in the pitch direction with respect to the traveling direction of the vehicle at the time when the cycle has elapsed and the angle of the orientation of the face at the time one cycle before that time, and the determination unit may determine whether the state of the driver is an abnormal state when a plurality of the calculated angular differences are less than the threshold value after the degree of probability has changed from the second degree to another degree during the state determination period.

[0015] The determination unit may re-determine whether the driver's state is abnormal based on a determination result of determining whether the direction of the driver's line of sight is in front of the traveling direction of the vehicle, on the condition that the degree of probability is a first degree indicating that the driver is wearing an item on the face, and the driver's state is determined to be abnormal because the direction of the face is included in the abnormal range.

[0016] The abnormal state has a plurality of types, and the determination unit determines an initial value of the state determination period that becomes larger for the abnormal state with a longer time for the determination unit to determine for each type of the abnormal state, and determines the state determination period for each type of the abnormal state by changing the initial value according to the degree of probability.

[0017] The driving support method according to the second aspect of the present invention includes a specifying step of specifying a degree of probability that the driver is wearing an item on the face based on feature points of the face extracted from a captured image generated by capturing an image of the face of the driver of the vehicle, which is executed by a computer, a state determination period for determining whether the driver's state is abnormal according to the degree of probability, and an abnormal range indicating the direction of the face when the driver's state is abnormal, and a determination step of determining whether the driver's state is abnormal by determining whether the direction of the face based on the feature points is included in the abnormal range during the state determination period.

Advantages of the Invention

[0018] According to the present invention, there is an effect of improving the accuracy of determining the state of a driver wearing an item on the face.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

MODE FOR CARRYING OUT THE INVENTION

[0020] <Overview of vehicle S> FIG. 1 is a diagram for explaining an overview of a vehicle S according to the present embodiment. The vehicle S shown in FIG. 1 includes an imaging device 1, a notification device 2, an input device 3, a vehicle control device 4, and a driving support device 10. The vehicle S has a function of determining whether the state of the driver of the vehicle S is an abnormal state. The abnormal state is a state in which the posture of the driver is lost and the driver cannot drive the vehicle S. For example, the posture of the driver is at least one of the states of lying prone, bowing, falling backward, bending like a shrimp, leaning sideways, and falling sideways.

[0021] The imaging device 1 is a camera capable of imaging a subject regardless of day or night, which is provided below the position of the face of the driver sitting on the driver's seat in the height direction and in front of the driver's seat in the traveling direction of the vehicle S. For example, it is an infrared camera. The imaging device 1 generates an imaging image of the face of the driver sitting on the driver's seat at a predetermined control cycle (for example, 0.1 second), and outputs it to the driving support device 10.

[0022] The notification device 2 is a device for notifying the driver that the driver is in an abnormal state. When the notification device 2 obtains state information indicating that the driver's state is an abnormal state from the driving support device 10, for example, the notification device 2 causes a warning sound to be emitted from a speaker it has, or causes a warning message to be displayed on a display it has. When the notification device 2 obtains operation information indicating that the input device 3 has received an operation for stopping the emission of the warning sound and the display of the warning message from the driving support device 10, for example, the notification device 2 stops the emission of the warning sound and the display of the warning message.

[0023] The input device 3 is a device for receiving the driver's operations, and includes, for example, operation buttons or a touch panel provided on at least one of the steering wheel and the instrument panel of the vehicle S. When the input device 3 receives an operation for stopping the emission of the warning sound and the display of the warning message from a driver who has recognized that his or her own state was an abnormal state, for example, the input device 3 outputs operation information indicating the operation to the driving support device 10.

[0024] The vehicle control device 4 is a device including, for example, one or more processors such as a CPU (Central Processing Unit) or an ECU (Electronic Control Unit). The vehicle control device 4 accelerates or decelerates the vehicle S according to the driver's operations, for example. When the vehicle control device 4 obtains instruction information indicating an instruction to stop the vehicle S from the driving support device 10, for example, the vehicle control device 4 stops the vehicle S.

[0025] The driving support device 10 executes a process for determining an abnormal state based on a captured image obtained from the imaging device 1 and generated by imaging the driver's face. When the driving support device 10 determines that the driver's state is an abnormal state, for example, the driving support device 10 causes the notification device 2 to start emitting a warning sound and displaying a warning message, or causes the vehicle control device 4 to stop the vehicle S. The driving support device 10 may have a housing including electronic components, or may be a printed circuit board on which the electronic components are mounted.

[0026] The driving support device 10 determines whether the driver's state is abnormal by, for example, identifying the orientation of the face based on the positions of the driver's eyes, mouth, nose, and the contour of the face (face line) extracted from the captured image, and determining whether the orientation of the face is within a predetermined range. The orientation of the face is, for example, the orientation in the pitch direction, yaw direction, and roll direction with respect to the traveling direction, and is indicated by the angle with the reference direction in each direction.

[0027] FIG. 2 is a diagram for explaining the orientation of the face of the driver D. FIG. 2(a) shows the orientation of the face in the pitch direction with respect to the traveling direction of the vehicle S, FIG. 2(b) shows the orientation of the face in the yaw direction with respect to the traveling direction of the vehicle S, and FIG. 2(c) shows the orientation of the face in the roll direction with respect to the traveling direction of the vehicle S. FIG. 2(c) is a view of the driver seen from the front in the traveling direction of the vehicle S toward the rear in the traveling direction. As shown in FIG. 2, the angles in the pitch direction, yaw direction, and roll direction with respect to the traveling direction of the vehicle S are represented with the direction in which the face of the driver D faces forward in the traveling direction as the reference direction (0 degrees). In the following description, the "pitch direction, yaw direction, roll direction with respect to the traveling direction of the vehicle S" may be simply referred to as the "pitch direction, yaw direction, roll direction".

[0028] By the way, the driver D may wear accessories such as a mask or an incubator on the face. FIG. 3 is a diagram showing an example of a driver wearing a mask M which is an accessory on the face. As shown in FIG. 3, when the driver D wears the mask M, even if the driving support device 10 can correctly identify the orientation of the face in the yaw direction and roll direction based on the position of the driver D's eyes, there is a possibility of incorrectly identifying the orientation of the face in the pitch direction by misrecognizing the unevenness of the mask M as the contour of the face.

[0029] Furthermore, driver D may wear the mask M with one of the nose and mouth uncovered, or wear a wearing item different from the mask M (for example, an in-come) simultaneously with the mask M. In this case, depending on the orientation of driver D's face, the driving support device 10 may correctly or incorrectly identify the orientation of the face in the pitch direction, resulting in a longer time until the orientation of the face in the pitch direction is correctly identified. As a result, even when driver D is driving in a correct posture, the driving support device 10 may erroneously determine that the state of driver D is an abnormal state.

[0030] Therefore, the driving support device 10 identifies the degree of probability that driver D is wearing a wearing item on the face. In the following description, the "degree of probability that the driver is wearing a wearing item on the face" may be simply referred to as the "degree of probability". The driving support device 10 identifies the first degree, the second degree, or the third degree as the degree of probability. The first degree indicates a state where, for example, the probability that driver D is wearing a wearing item on the face is estimated to be 100%. The second degree indicates a state where, for example, there is a possibility that driver D is wearing a wearing item on the face (that is, the probability that the driver is wearing a wearing item on the face is greater than 0 and less than 100%). The third degree indicates a state where the probability that driver D is wearing a wearing item on the face is estimated to be 0%.

[0031] Then, the driving support device 10 determines a state determination period for determining an abnormal state and an abnormal range of the face orientation for determining that the driver's state is an abnormal state according to the degree of probability. For example, when the degree of probability is the first degree, the driving support device 10 makes the abnormal range smaller than when the degree of probability is other degrees. By operating in this way, even if the driving support device 10 misidentifies the unevenness of the wearing item as the contour of the face, it can suppress erroneously determining the state of driver D due to the small abnormal range.

[0032] Furthermore, for example, when the degree of probability is the second degree, the driving support device 10 makes the state determination period longer than when the degree of probability is other degrees. By operating in this way, the driving support device 10 can correctly determine the state of the driver D even when the time for correctly identifying the face direction according to the state in which the driver D is wearing the clothing becomes longer. As a result of the above, the driving support device 10 can improve the accuracy of determining whether the state of the driver D is an abnormal state. Hereinafter, the configuration and operation of the driving support device 10 will be described in detail.

[0033] <Configuration of Driving Support Device 10> As shown in FIG. 1, the driving support device 10 includes a storage unit 11 and a control unit 12. The control unit 12 includes an acquisition unit 121, an extraction unit 122, a specification unit 123, a determination unit 124, a determination unit 125, and an instruction unit 126.

[0034] The storage unit 11 has a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive). The storage unit 11 stores a program executed by the control unit 12. The storage unit 11 stores various information for determining an abnormal state.

[0035] The control unit 12 is a processor such as a CPU or an ECU, for example. The control unit 12 functions as an acquisition unit 121, an extraction unit 122, a specification unit 123, a determination unit 124, a determination unit 125, and an instruction unit 126 by executing the program stored in the storage unit 11. Note that the control unit 12 may be configured by one processor, or may be configured by a combination of a plurality of processors or one or more processors and an electronic circuit. Hereinafter, the configuration of each unit realized by the control unit 12 will be described.

[0036] The acquisition unit 121 acquires the captured image generated by capturing the face of the driver D of the vehicle S from the imaging device 1. The acquisition unit 121 acquires the captured image, for example, at a predetermined control period. The predetermined control period is, for example, 0.1 second. The extraction unit 122 extracts facial feature points (hereinafter referred to as "facial feature points") from the captured image acquired by the acquisition unit 121. The facial feature points include, for example, the positions of the components of the face such as eyebrows, eyes, nose, and mouth, the position of the facial contour, and the direction of the line of sight. The extraction unit 122 extracts the facial feature points, for example, at a predetermined control period.

[0037] The specifying unit 123 specifies the position and orientation of the face of the driver D based on the facial feature points extracted from the captured image by the extraction unit 122. The specifying unit 123 specifies the position of the face of the driver D, for example, based on the position of the facial contour included in the facial feature points. The specifying unit 123 specifies the orientation of the face in the pitch direction with respect to the traveling direction of the vehicle S, for example, based on the position of the facial contour and the position of the mouth included in the facial feature points. The specifying unit 123 specifies the orientation of the face in the yaw direction and the roll direction with respect to the traveling direction of the vehicle S, for example, based on the position of the eyes included in the facial feature points.

[0038] The specifying unit 123 specifies the degree of probability that the driver D is wearing an item on the face based on the facial feature points extracted from the captured image by the extraction unit 122. The specifying unit 123 determines, for example, whether or not the mouth feature points indicating the mouth of the face are included in the facial feature points extracted by the extraction unit 122 at a predetermined control period. Then, when it is determined that the mouth feature points are not included in the wearing determination period including a plurality of predetermined control periods, the specifying unit 123 specifies the first degree indicating that the driver D is wearing an item on the face as the degree of probability. The wearing determination period is, for example, a period of 0.2 seconds or more and 0.5 seconds or less when the control period is 0.1 second.

[0039] FIG. 4 is a diagram for explaining an operation of specifying a degree of probability. The horizontal axis in FIG. 4 indicates time, and the vertical axis in FIG. 4 indicates "mouth feature points", which is the result of determining whether or not mouth feature points are included in face feature points, and "face orientation (pitch direction)", which is the result of specifying the orientation of the face in the pitch direction with respect to the traveling direction of vehicle S. The "face orientation (pitch direction)" shown in FIG. 4 indicates a positive value when the face orientation of driver D is upward, and a negative value when the face orientation of driver D is downward. The specifying unit 123, for example, determines that no mouth feature points are included in the face feature points extracted by the extraction unit 122 from a plurality of captured images during the wearing determination period P11 from time T11 to time T12, and specifies the degree of probability as the first degree.

[0040] The specifying unit 123, for example, when making a determination that mouth feature points are included in the face feature points and a determination that mouth feature points are not included in the face feature points during the wearing determination period, specifies the second degree, which indicates that there is a possibility that driver D is wearing an item on the face, as the degree of probability. The state of "there is a possibility that driver D is wearing an item on the face" is a state where it is suspected that an item is being worn but it cannot be determined. This state includes, for example, cases where driver D is wearing an item but the nose or mouth is not covered, cases where driver D is wearing a plurality of items, or cases where the shape of the item is similar to the shape of the unevenness of the face and it is easy to misrecognize the item as a component of the face (for example, the nose). The specifying unit 123, for example, during the wearing determination period P12 from time T12 to time T14 shown in FIG. 4, determines that no mouth feature points are included at times before time T13, and determines that mouth feature points are included at times after time T13. Then, the specifying unit 123 specifies the degree of probability as the second degree.

[0041] The specifying unit 123 may specify the degree of probability as the second degree based on the orientation of the face in the pitch direction with respect to the traveling direction of vehicle S. The specifying unit 123, for example, determines whether the orientation of the face in the pitch direction with respect to the traveling direction of vehicle S is upward or downward at a predetermined control cycle. Then, the specifying unit 123, for example, when the upward determination and the downward determination are alternately repeated during the wearing determination period, specifies the second degree as the degree of probability.

[0042] For example, in the wearing determination period P13 from time T14 to time T15 shown in FIG. 4, the specific unit 123 specifies that the face orientation in the pitch direction with respect to the traveling direction of the vehicle S alternates between upward and downward (so-called hunting occurs). Then, for example, even if the specific unit 123 determines that the mouth feature points are included in each captured image during the wearing determination period P13, due to the occurrence of hunting, the degree of probability is specified as the second degree. By operating in this way, the specific unit 123 can estimate whether there is an error in the face feature points based on the face orientation, even if the extraction unit 122 erroneously extracts the face feature points, so that it can be suppressed to specify the degree of probability erroneously.

[0043] For example, during the wearing determination period, if each face feature point extracted from each captured image includes mouth feature points and the face orientation in the pitch direction with respect to the traveling direction of the vehicle S does not alternate between upward and downward, the specific unit 123 specifies the third degree as the degree of probability. The third degree indicates the degree that the driver D is not wearing an item on the face.

[0044] The determination unit 124 determines an abnormal range indicating the face orientation when the state of the driver D is an abnormal state according to the degree of probability specified by the specific unit 123. In the abnormal range, an initial range is defined in each of the pitch direction, yaw direction, and roll direction with respect to the traveling direction of the vehicle S and is stored in the storage unit 11. For example, when the degree of probability is the second degree or the third degree, the determination unit 124 determines the initial range of the abnormal range stored in the storage unit 11 as the abnormal range.

[0045] When the degree of probability is the first degree, for example, the determination unit 124 makes the range of the face orientation in the pitch direction with respect to the traveling direction of the vehicle S indicated by the abnormal range smaller than when the degree of probability is a degree different from the first degree (that is, the second degree or the third degree). For example, when the degree of probability is the first degree, the determination unit 124 determines, as the abnormal range in the pitch direction, a range smaller than the range of the face orientation in the pitch direction with respect to the traveling direction of the vehicle S included in the initial range of the abnormal range. Specifically, when the initial range of the abnormal range in the pitch direction with respect to the traveling direction of the vehicle S is "-90 degrees or more and less than -20 degrees", the determination unit 124 determines the abnormal range in the pitch direction to be "-90 degrees or more and less than -30 degrees" because the degree of probability is the first degree.

[0046] By operating in this way, the determination unit 124 can reduce the range in which the determination unit 125 determines that the state of the driver D is an abnormal state, so that it is possible to suppress misjudging the state of the driver D based on the face orientation erroneously specified by the specifying unit 123. When the degree of probability is the first degree, the determination unit 124 determines the abnormal range of the face orientation in the yaw direction and the roll direction with respect to the traveling direction of the vehicle S to the initial range stored in the storage unit 11.

[0047] The determination unit 124 determines a state determination period for determining whether or not the state of the driver D is an abnormal state according to the degree of probability specified by the specifying unit 123. The initial value of the state determination period is, for example, a period of 3 seconds or more and 7 seconds or less, and is stored in the storage unit 11. For example, when the degree of probability is the first degree or the third degree, the determination unit 124 determines the initial value of the state determination period stored in the storage unit 11 as the state determination period.

[0048] For example, when the degree of probability is the second degree, the determination unit 124 makes the state determination period longer than when the degree of probability is different from the second degree (i.e., the first degree or the third degree). For example, when the degree of probability is the second degree, the determination unit 124 determines the state determination period as the period obtained by multiplying the initial value of the state determination period stored in the storage unit 11 by 2. By operating in this way, after the identification unit 123 switches from the state of erroneously identifying the face orientation to the state of correctly identifying the face orientation, the determination unit 125 can execute the abnormal state determination, so that it is possible to suppress misjudging the state of the driver D.

[0049] The determination unit 124 may determine the state determination period based on the angle of the face orientation identified by the identification unit 123 and the degree of probability. For example, during the state determination period, for each predetermined control cycle, the determination unit 124 calculates the angle difference between the angle of the face orientation in the pitch direction with respect to the traveling direction of the vehicle S at the time when the predetermined control cycle has elapsed and the angle of the face orientation at the time before the predetermined control cycle from the said time. Then, for example, when the degree of probability identified by the identification unit 123 is the second degree and the calculated angle difference is equal to or greater than the threshold value, the determination unit 124 makes the state determination period longer. The threshold value is the minimum value of the angle difference that can be estimated that the identification unit 123 has erroneously identified the face orientation due to wearing an article on the face, and is a value determined by experiments or simulations.

[0050] FIG. 5 is a diagram for explaining the operation of determining the state determination period. The horizontal axis of FIG. 5 represents time. The vertical axis of FIG. 5 represents the "degree of probability" identified by the identification unit 123 for each wearing determination period, the "face orientation (pitch direction)" indicating the angle of the face orientation in the pitch direction with respect to the traveling direction of the vehicle S, and the "abnormal determination" indicating the timing of executing the abnormal state determination. The "×" shown in FIG. 5 is the timing when the execution of the abnormal state determination is aborted, and the "○" shown in FIG. 5 is the timing when the abnormal state determination is executed.

[0051] For example, the determination unit 124 specifies that the angular difference between the face orientation angle D2 at time T23 and the face orientation angle D1 at time T22, which is a predetermined control cycle before time T23, is equal to or greater than a threshold value, and that the degree of probability from time T22 to time T23 is the second degree. Then, at time T23, the determination unit 124 changes the state determination period from period P21 to period P22, thereby lengthening the state determination period. Similarly to the above, the determination unit 124 specifies that the angular difference between the face orientation angle D4 at time T26 and the face orientation angle D3 at time T25 is equal to or greater than a threshold value, and that the degree of probability from time T25 to time T26 is the second degree. Then, at time T26, the determination unit 124 further changes the state determination period from period P22 to period P23.

[0052] Subsequently, since the angular difference calculated by the determination unit 124 from time T27 to time T28 is less than the threshold value, the determination unit 124 determines a new state determination period at time T28 without changing the state determination period from period P23. By operating as described above, the determination unit 124 can extend the period including the time when the identification unit 123 erroneously identifies the face orientation of the driver D. As a result, the determination unit 125 can perform an abnormal state determination at the time when the identification unit 123 correctly identifies the face orientation within the state determination period determined by the determination unit 124, so that the accuracy of the abnormal state determination can be improved. Note that the period P21, the difference between period P22 and period P21, and the difference between period P23 and period P22 determined by the determination unit 124 may be the same or different.

[0053] When the calculated angular difference is equal to or greater than the threshold value and the degree of probability is the first degree, the determination unit 124 may lengthen the state determination period. By operating in this way, the determination unit 124 can lengthen the time for determining whether it is an abnormal state when the driver D is wearing an item on the face. As a result, the probability that the determination unit 125 performs an abnormal state determination at the time when the identification unit 123 erroneously identifies the face orientation can be reduced, so that the accuracy of the abnormal state determination can be improved.

[0054] During the state determination period, the determination unit 125 performs an abnormal state determination as to whether the state of the driver D is an abnormal state by determining whether the face orientation based on the face feature points is included in the abnormal range. "The face orientation is included in the abnormal range" means that at least one of the face orientations in the pitch direction, yaw direction, and roll direction of the face with respect to the traveling direction is included in the abnormal range corresponding to the face orientation.

[0055] For example, the determination unit 125 determines whether the face orientation specified by the specifying unit 123 is included in the abnormal range determined by the determining unit 124 for each predetermined control cycle included in the state determination period. Then, for example, when the determination unit 125 determines that the face orientation is included in the abnormal range in a predetermined ratio or more of the plurality of control cycles included in the state determination period, the determination unit 125 determines that the state of the driver D is an abnormal state. On the other hand, for example, when the determination unit 125 determines that the face orientation is included in the abnormal range in less than a predetermined ratio of the plurality of control cycles included in the state determination period, the determination unit 125 determines that the state of the driver D is not an abnormal state. The predetermined ratio is a ratio determined by experiment or simulation.

[0056] For example, when the degree of probability specified by the specifying unit 123 is the first degree or the third degree, the determination unit 125 determines whether the face orientation specified by the specifying unit 123 is included in the abnormal range determined by the determining unit 124 according to the degree of probability. For example, when the degree of probability is the second degree, the determination unit 125 determines whether the face orientation specified by the specifying unit 123 is included in the abnormal range corresponding to the other degree after the degree of probability changes from the second degree to the other degree. By operating in this way, the determination unit 125 can suppress performing the abnormal state determination in a state where it is ambiguous whether the driver D is wearing an item on the face.

[0057] During the state determination period, if the angular differences of the orientations of a plurality of faces calculated after the degree of probability changes from the second degree to another degree are less than the threshold value, the determination unit 125 may determine whether the state of the driver D is an abnormal state. For example, the determination unit 124 calculates the angular difference of the face orientation in the pitch direction with respect to the traveling direction of the vehicle S after the degree of probability changes from the second degree to the first degree during the time from time T27 to time T28 included in the state determination period P23 shown in FIG. 5. Then, during the state determination period P23, the determination unit 125 determines whether the face orientation specified by the specifying unit 123 after time T27 is included in the abnormal range corresponding to the first degree determined by the determination unit 124 because the angular difference of the face orientation calculated after time T27 is less than the threshold value.

[0058] By operating as described above, the determination unit 125 can suppress executing the abnormal state determination in a state where it is ambiguous whether the driver D is wearing an item on the face. Furthermore, the determination unit 125 can suppress executing the abnormal state determination in a state where the specifying unit 123 has erroneously specified the orientation of the driver D's face. As a result, the determination unit 125 can improve the accuracy of the abnormal state determination.

[0059] Incidentally, even if the specifying unit 123 erroneously specifies the posture of the driver D (that is, the position and orientation of the face) because the driver D is wearing an item and the mouth feature points are not included in the face feature points, it may be possible to correctly specify the direction of the driver D's line of sight included in the face feature points. Therefore, after executing the abnormal state determination based on the orientation of the driver D's face, the determination unit 125 may re-determine whether the state of the driver D is an abnormal state based on the direction of the driver D's line of sight.

[0060] The determination unit 125 re-determines whether the state of the driver D is an abnormal state on the condition that, for example, the degree of probability is the first degree and the face orientation is included in the abnormal range, thus determining that the state of the driver D is an abnormal state. The determination unit 125 re-determines whether the state of the driver D is an abnormal state based on, for example, the determination result of determining whether the direction of the line of sight of the driver D included in the face feature points is in front of the traveling direction of the vehicle S. For example, when the direction of the line of sight of the driver D is in front of the traveling direction of the vehicle S, the determination unit 125 re-determines that the state of the driver D is not an abnormal state; when the direction of the line of sight of the driver D is in a direction different from the front of the traveling direction of the vehicle S, the determination unit 125 re-determines that the state of the driver D is an abnormal state.

[0061] For example, if the result of the abnormal state determination before re-determination is different from the result of the re-determination, the determination unit 125 uses the result of the re-determination as the determination result. By operating in this way, the determination unit 125 can accurately determine whether the state of the driver D is an abnormal state even when the mouth feature points are not included in the face feature points. Also, when the degree of probability is the first degree, the determination unit 125 does not have to determine whether the state of the driver D is an abnormal state. By operating in this way, the determination unit 125 can reduce the probability of misjudging the state of the driver D.

[0062] Based on the determination result determined by the determination unit 125, the instruction unit 126 notifies the driver D that the state of the driver D is abnormal or stops the vehicle S. For example, when the determination unit 125 determines that the state of the driver D is an abnormal state, the instruction unit 126 outputs state information indicating that the state of the driver D is an abnormal state to the notification device 2, causing a warning sound to be emitted from the speaker of the notification device 2. For example, by outputting the state information to the notification device 2, the instruction unit 126 causes a warning message to be displayed on the display of the notification device 2.

[0063] When the instruction unit 126 acquires operation information indicating that the driver D has performed an operation to stop the warning sound and the warning message within a predetermined time from the time when the status information was output from the input device 3, the notification device 2 is caused to stop the output of the warning sound and the display of the warning message. When the instruction unit 126 does not acquire the operation information within the predetermined time, the instruction unit 126 outputs instruction information indicating an instruction to stop the vehicle S to the vehicle control device 4, thereby stopping the vehicle S.

[0064] <Processing sequence in the driving support device 10> FIG. 6 and FIG. 7 are diagrams showing examples of processing sequences in the driving support device 10. The processing sequence shown in FIG. 6 is a sequence showing an operation in which the specifying unit 123 specifies the degree of probability, and the processing sequence shown in FIG. 7 is a sequence showing an operation in which the determination unit 124 determines the abnormal range and the state determination period.

[0065] First, the operation in which the specifying unit 123 specifies the degree of probability will be described with reference to FIG. 6. The extraction unit 122 extracts face feature points from the captured image acquired by the acquisition unit 121 (S11). The specifying unit 123 specifies the mouth feature points included in the face feature points extracted by the extraction unit 122 (S12). The specifying unit 123 specifies the orientation of the face based on the face feature points (S13). When the wearing determination period has not elapsed (NO in S14), the acquisition unit 121, the extraction unit 122, and the specifying unit 123 repeat the operations from step S11 to step S13.

[0066] When the wearing determination period has elapsed (YES in S14), the specifying unit 123 determines whether or not mouth feature points are included for each face feature point extracted by the extraction unit 122 during the wearing determination period. If no mouth feature point is included in each face feature point extracted during the wearing determination period (YES in S15), that is, if no mouth feature point is included in all of the face feature points extracted during the wearing determination period, the specifying unit 123 specifies the first degree indicating that the driver D is wearing an article on the face in terms of the degree of probability (S16). If a mouth feature point is included in one or more of the plurality of face feature points extracted during the wearing determination period (NO in S15), the specifying unit 123 determines whether or not a mouth feature point is included in each face feature point (that is, whether or not a mouth feature point is included in all of the face feature points extracted during the wearing determination period).

[0067] If a mouth feature point is included in each face feature point extracted during the wearing determination period (YES in S17), that is, if a mouth feature point is included in all of the face feature points extracted during the wearing determination period, the specifying unit 123 specifies the third degree indicating that the driver D is not wearing an article on the face in terms of the degree of probability (S18). In S17, if a mouth feature point is not included in one or more of the plurality of face feature points extracted during the wearing determination period (NO in S17), that is, if both face feature points including a mouth feature point and face feature points not including a mouth feature point are included in the plurality of face feature points extracted during the wearing determination period, the specifying unit 123 specifies the second degree indicating that there is a possibility that the driver D is wearing an article on the face in terms of the degree of probability (S19).

[0068] Next, the operation in which the determination unit 124 determines the abnormal range and the state determination period will be described with reference to FIG. 7. The determination unit 124 refers to the initial value of the abnormal range and the initial range of the state determination period stored in the storage unit 11, determines the initial value as the abnormal range, and determines the initial range as the state determination period (S21). When the degree of probability specified by the specifying unit 123 is the first degree (YES in S22), the determination unit 124 changes the abnormal range by reducing the range of the face orientation in the pitch direction with respect to the traveling direction of the vehicle S indicated by the abnormal range (S23). When the degree of probability specified by the specifying unit 123 is not the first degree (NO in S22), the determination unit 124 determines whether the degree of probability is the second degree (S24).

[0069] When the degree of probability is not the first degree and is not the second degree (NO in S24), that is, when the degree of probability is the third degree, the determination unit 124 ends the process. When the degree of probability is the second degree (YES in S24), the determination unit 124 calculates an angular difference during the state determination period based on the angle of the face orientation specified by the specifying unit 123 (S25). The angular difference is the difference between the angle of the face orientation in the pitch direction with respect to the traveling direction of the vehicle S at the time when a predetermined control cycle has elapsed and the angle of the face orientation at the time before the predetermined control cycle from that time. When the angular difference calculated by the determination unit 124 is equal to or greater than the threshold value (YES in S26), the determination unit 124 extends the state determination period (S27). When the angular difference calculated by the determination unit 124 is less than the threshold value (NO in S26), the determination unit 124 ends the process.

[0070] <Modification example> In the above description, the operation of determining one state determination period for determining whether the state of the driver D is an abnormal state has been illustrated, but the present invention is not limited to this. When there are a plurality of types of abnormal states, the determination unit 124 may determine a state determination period for each type of abnormal state.

[0071] FIG. 8 is a diagram showing the initial values of the state determination periods according to the modification. In FIG. 8, the types of abnormal states and the initial ranges of the state determination periods corresponding to the types of each abnormal state are shown. As shown in FIG. 8, when the type of abnormal state is "lying down", since the face direction and the line-of-sight direction are likely to be the same, the state determination period can be shortened. On the other hand, when the type of abnormal state is "bowed head", since the face direction and the line-of-sight direction may be different, the time for re-determination becomes large and the state determination period becomes long. Note that the initial ranges of the state determination periods shown in FIG. 8 may be stored in the storage unit 11.

[0072] The determination unit 124 determines, for example, the initial value (initial range) of the state determination period for each type of abnormal state. The determination unit 124 determines, for example, the initial value (initial range) of the state determination period to be larger for an abnormal state with a longer time for the determination unit 125 to make a determination. Then, the determination unit 124 determines the state determination period for each type of abnormal state by changing the initial value (initial range) according to, for example, the degree of probability.

[0073] Specifically, the determination unit 124 refers to the storage unit 11 to determine a first state determination period of "3 seconds" corresponding to "lying down" and a second state determination period of "7 seconds" corresponding to "bowed head". Then, for example, when the degree of probability is the second degree, the determination unit 124 determines "6 seconds", which is obtained by integrating 2 into the first state determination period, as the new first state determination period, and determines "14 seconds", which is obtained by integrating 2 into the second state determination period, as the new second state determination period. By operating in this way, the determination unit 124 can determine a state determination period suitable for the type of abnormal state.

[0074] <Effect of the driving support device 10> As described above, the driving support device 10 includes a specifying unit 123 that specifies the degree of probability that the driver D of the vehicle S is wearing an item (mask) on the face based on the feature points of the face extracted from the captured image generated by capturing the face of the driver D; a state determination period that determines whether or not the state of the driver D is an abnormal state according to the degree of probability; and an abnormal range that indicates the orientation of the face when the state of the driver D is an abnormal state. The driving support device 10 further includes a determination unit 125 that determines whether or not the state of the driver D is an abnormal state by determining whether or not the orientation of the face based on the feature points is included in the abnormal range during the determined state determination period.

[0075] With the driving support device 10 configured in this way, the driving support device 10 can determine the abnormal range and the state determination period according to the degree of probability that the driver D is wearing an item. By reducing the abnormal range, even if the unevenness of the item is erroneously identified as the contour of the face, the probability of erroneously determining the state of the driver D can be reduced due to the small abnormal range. Further, by lengthening the state determination period, the driving support device 10 can determine the state of the driver D after the orientation of the face has changed from a state where the orientation of the face is erroneously identified to a state where the orientation of the face can be correctly identified. As a result, the driving support device 10 can determine with high accuracy whether or not the state of the driver D is an abnormal state.

[0076] As described above, the present invention has been described using the embodiments. However, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist thereof. For example, all or part of the device can be configured by being functionally or physically dispersed and integrated in an arbitrary unit. Also, new embodiments resulting from an arbitrary combination of a plurality of embodiments are included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination have the effects of the original embodiments combined.

Description of Reference Numerals

[0077] 1 Imaging device 2 Notification device 3 Input device 4 Vehicle control device 10 Driving support device 11 Memory unit 12 Control unit 121 Acquisition unit 122 Extraction unit 123 Identification unit 124 Decision unit 125 Judgment unit 126 Instruction unit

Claims

1. A specifying unit that specifies the degree of probability that the driver is wearing an item on the face based on the feature points of the face extracted from the captured image generated by capturing the face of the vehicle driver; A determination unit that determines a state determination period for determining whether the state of the driver is an abnormal state according to the degree of probability, and an abnormal range indicating the orientation of the face when the state of the driver is an abnormal state; A determination unit that determines whether the state of the driver is an abnormal state by determining whether the orientation of the face based on the feature points is included in the abnormal range during the state determination period; A driving support device having the above.

2. The specifying unit determines, at a predetermined cycle, whether the extracted feature points include a mouth feature point indicating the mouth of the face, and when it is determined that the mouth feature point is not included during a wearing determination period including a plurality of the predetermined cycles, specifies the first degree indicating that the driver is wearing an item on the face as the degree of probability; The driving support device according to Claim 1.

3. The specifying unit determines, at a predetermined cycle, whether the extracted feature points include a mouth feature point indicating the mouth of the face, and when both the determination that the mouth feature point is included and the determination that the mouth feature point is not included are made during a wearing determination period including a plurality of the predetermined cycles, specifies the second degree indicating that there is a possibility that the driver is wearing an item on the face as the degree of probability; The driving support device according to Claim 1.

4. The specifying unit determines, at a predetermined cycle, whether the orientation of the face in the pitch direction with respect to the traveling direction of the vehicle is upward or downward, and when the upward determination and the downward determination alternate during a wearing determination period including a plurality of the predetermined cycles, specifies the second degree indicating that there is a possibility that the driver is wearing an item on the face as the degree of probability; The driving support device according to Claim 1.

5. When the degree of probability is the first degree, the determination unit makes the range of the orientation of the face in the pitch direction with respect to the traveling direction of the vehicle indicated by the abnormal range smaller than when the degree of probability is a degree different from the first degree; The driving support device according to Claim 2.

6. When the degree of probability is the first degree, the determination unit does not determine whether the state of the driver is an abnormal state; The driving assistance device according to claim 2.

7. When the degree of probability is the second degree, the determination unit makes the state determination period longer than when the degree of probability is different from the second degree. The driving assistance device according to claim 3 or 4.

8. In each cycle during the state determination period, the determination unit calculates the angle difference between the angle of the face in the pitch direction with respect to the traveling direction of the vehicle at the time when the cycle has elapsed and the angle of the face at the time before the cycle from that time, and when the degree of probability is the second degree and the angle difference is greater than or equal to the threshold value, the determination unit makes the state determination period longer. The driving assistance device according to claim 3 or 4.

9. In each cycle during the state determination period, the determination unit calculates the angle difference between the angle of the face in the pitch direction with respect to the traveling direction of the vehicle at the time when the cycle has elapsed and the angle of the face at the time before the cycle from that time. During the state determination period, when a plurality of the calculated angle differences after the degree of probability has changed from the second degree to another degree are less than the threshold value, the determination unit determines whether the state of the driver is an abnormal state. The driving assistance device according to claim 3 or 4.

10. Based on the determination result of determining whether the direction of the driver's line of sight is in front of the traveling direction of the vehicle, on the condition that the degree of probability is the first degree indicating that the driver is wearing an object on the face and the direction of the face is included in the abnormal range, and it has been determined that the state of the driver is an abnormal state, the determination unit re-determines whether the state of the driver is an abnormal state. The driving assistance device according to claim 1.

11. The abnormal state has a plurality of types. For each type of abnormal state, the determination unit determines the initial value of the state determination period, which becomes larger for the abnormal state with a longer time for the determination unit to make a determination, and determines the state determination period for each type of abnormal state by changing the initial value according to the degree of probability. The driving assistance device according to claim 1.

12. Executed by a computer A specifying step of specifying the degree of probability that the driver is wearing an article on the face based on the feature points of the face extracted from the captured image generated by capturing the face of the driver of the vehicle; A determining step of determining a state determination period for determining whether the state of the driver is an abnormal state according to the degree of probability, and an abnormal range indicating the orientation of the face when the state of the driver is an abnormal state; A determining step of determining whether the state of the driver is an abnormal state by determining whether the orientation of the face based on the feature points is included in the abnormal range during the state determination period; A driving support method having the above.

Images