Drowsiness estimation device and drowsiness estimation method

Abstract

The present invention provides a drowsiness estimation device comprising: a drowsiness score calculation unit (15) that uses a trained machine learning model to calculate a drowsiness score indicating a degree of drowsiness of an occupant of a vehicle on the basis of a captured image of the face of the occupant; an excessive detection factor reliability calculation unit (16) that determines whether an excessive detection factor affecting the calculation of the drowsiness score is present on the basis of the image, and, upon determining that the excessive detection factor exists, calculates, on the basis of a rule, an excessive detection factor indicator indicating the existence of the excessive detection factor and reliability of the excessive detection factor indicator; and a state transition determination unit (17) that determines, on the basis of the calculated drowsiness score, the calculated excessive detection factor indicator, and the calculated reliability of the excessive detection factor indicator, a drowsiness state of the occupant from a plurality of drowsiness states including a first drowsiness state in which the degree of drowsiness is low and a second drowsiness state in which the degree of drowsiness is higher than that in the first drowsiness state.

Description

[Technical field]

[0001] The present disclosure relates to drowsiness estimation technology. [Background technology]

[0002] One of the causes of traffic accidents is driving errors caused by driver drowsiness. In order to prevent such traffic accidents, a drowsiness determination function is sometimes used as one of the functions of a passenger monitoring system (hereinafter sometimes simply referred to as "PMS") that monitors the vehicle's driver and other passengers. The drowsiness determination function is a function that captures information about passengers using sensors such as cameras and estimates their level of drowsiness. Traffic accidents are prevented by issuing a warning to wake up the passengers based on the estimation results.

[0003] To realize the drowsiness determination function, the characteristics of the occupant, such as closing the eyes or yawning, are quantified as features (hereinafter referred to as "drowsiness features"), and the presence or absence of drowsiness and the degree of drowsiness are determined by comprehensively judging the changes in the quantified drowsiness features. However, because there are large individual differences in how drowsiness is expressed in the drowsiness features when drowsiness occurs, it is difficult to determine drowsiness using a rule-based model by formulating rules for the cases in which drowsiness is determined to be present.

[0004] In order to solve the difficulties associated with such a rule-based model, there is a technique for determining drowsiness using a machine learning model, such as the technique disclosed in Patent Document 1. In addition, paragraph 0020 of Patent Document 1 states that the machine learning model and the rule-based model may be combined. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2020-514861 Summary of the Invention [Problem to be solved by the invention]

[0006] The value of the drowsiness feature may change in the same way when the driver is drowsy and when the driver is not drowsy. Therefore, it is difficult to distinguish between these cases by a technique that simply uses a machine learning model to determine drowsiness. As described above, although Patent Document 1 describes that a machine learning model and a rule-based model may be combined, it does not mention a specific method of combination. Therefore, the conventional technique has a problem that it is not clear how to estimate drowsiness by combining a machine learning model and a rule-based model.

[0007] The present disclosure has been made to solve such problems, and aims to provide a drowsiness estimation technology that estimates the drowsiness of a vehicle occupant by combining a machine learning model and a rule-based model. [Means for solving the problem]

[0008] A drowsiness estimation device according to one aspect of an embodiment of the present disclosure includes a drowsiness score calculation unit that calculates a drowsiness score indicating the level of drowsiness of a vehicle occupant using a trained machine learning model based on an image of the face of the vehicle occupant; an overdetection factor reliability calculation unit that determines whether there are overdetection factors that affect the calculation of the drowsiness score based on the image, and when it is determined that the overdetection factor exists, calculates an overdetection factor index indicating the presence of the overdetection factor and a reliability of the overdetection factor index in a rule-based manner; and a state transition determination unit that determines which of a plurality of drowsiness states the occupant is in, including a first drowsiness state having a lower level of drowsiness and a second drowsiness state having a higher level of drowsiness than the first drowsiness state, based on the calculated drowsiness score, the calculated overdetection factor index, and the reliability of the calculated overdetection factor index. Effect of the Invention

[0009] According to the drowsiness estimation device according to an embodiment of the present disclosure, the drowsiness of a vehicle occupant can be estimated by combining a machine learning model and a rule-based model. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of the configuration of a drowsiness estimation device. [Figure 2A] FIG. 2 is a diagram illustrating an example of a hardware configuration of a drowsiness estimation device. [Figure 2B] FIG. 2 is a diagram illustrating an example of a hardware configuration of a drowsiness estimation device. [Diagram 3] 4 is a flowchart showing a schematic operation of the drowsiness estimation device. [Figure 4] 10 is a flowchart showing an operation of an overdetection factor reliability calculation unit of the drowsiness estimation device. [Diagram 5] FIG. 13 is a diagram for explaining a method for calculating a rate at which both eyelids are open. [Figure 6] 5 is a flowchart showing an operation of a state transition determination unit of the drowsiness estimation device. [Figure 7] FIG. 2 is a state transition diagram between a plurality of drowsiness states. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Various embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that components with the same or similar reference numerals in the drawings have the same or similar configurations or functions, and redundant descriptions of such components will be omitted.

[0012] Embodiment 1 <Configuration> A drowsiness estimation device according to the first embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of a drowsiness estimation device 100 according to the first embodiment of the present disclosure. As shown in Fig. 1, a vehicle 1 includes an imaging device 2 and the drowsiness estimation device 100.

[0013] (Imaging device) The imaging device 2 is a device for capturing an image of the interior of the vehicle 1. The imaging device 2 is installed, for example, in the front part of the interior of the vehicle 1, and captures an image of an area including the face of an occupant such as a driver of the vehicle 1 from the front. The imaging device 2 is configured with one visible light camera, multiple visible light cameras, one infrared camera, or multiple infrared cameras. When the imaging device 2 is configured with an infrared camera, a light source (not shown) is provided that irradiates an area including the driver's face with infrared light for capturing an image. This light source is configured with, for example, an LED (Light Emitting Diode).

[0014] (Drowsiness estimation device) As shown in FIG. 1, the drowsiness estimation device 100 includes a face information detection unit 10 and a drowsiness determination unit 20. The face information detection unit 10 includes an image acquisition unit 11, a face detection unit 12, and a face part detection unit 13. The drowsiness determination unit 20 includes a feature amount calculation unit 14, a drowsiness score calculation unit 15, an overdetection factor reliability calculation unit 16, and a state transition determination unit 17. The drowsiness estimation device 100 also includes a control unit (not shown) (hereinafter simply referred to as a "control unit"). In FIG. 1, the control unit is not shown in order to show the overall flow of data. The control unit controls the operation of the imaging device 2 and each functional unit of the drowsiness estimation device 100. Specifically, the control unit issues an instruction to the imaging device 2 to capture an image of the driver with an in-vehicle camera. In addition, the control unit issues instructions to the image acquisition unit 11, the face detection unit 12, the facial part detection unit 13, the feature calculation unit 14, the drowsiness score calculation unit 15, the overdetection factor reliability calculation unit 16, and the state transition determination unit 17 to control the operation timing and the exchange of information.

[0015] (Video acquisition section) The video acquisition unit 11 acquires a facial image from a video (imaging information) including a plurality of frame images captured by the imaging device 2, and outputs the acquired facial image to the control unit. A facial image means an image in which a partial area including a face is extracted from the entire area of ​​each frame image.

[0016] (Face detection section) The face detection unit 12 receives the face image acquired by the video acquisition unit 11 from the control unit, detects a face from the face image, and outputs the detected face data as a face detection result to the control unit. The face detection unit 12 is a classifier using a general algorithm, for example, a combination of a Haar-Like detector with Adaboost or Casecade.

[0017] (Facial feature detection section) The facial part detection unit 13 receives the face detection result output by the face detection unit 12 from the control unit, detects facial parts such as the eyes or mouth from the face data included in the face detection result, and outputs the detected facial parts to the control unit as the facial part detection result. The facial parts can be detected using a general image processing technique, for example, by detecting edges using a differential filter.

[0018] In addition, the face part detection unit 13 may calculate the degree of opening of the face part from the shape of the detected face part, and may output information indicating the degree of opening of the face part by including it in the face part detection result. For example, the degree of opening of the eyes may be calculated as information indicating the degree of opening of the face part. The degree of opening of the eyes may be calculated by detecting the positions of the inner corner of the eye, the outer corner of the eye, and the highest point of the upper eyelid, dividing the distance between the straight line connecting the inner corner and the outer corner of the eye and the highest point of the upper eyelid by the distance between the inner corner and the outer corner of the eye to obtain the flattening of the eyes, and dividing the flattening by a predetermined reference value. Note that the highest point of the upper eyelid is the point (vertex) of the upper eyelid that is farthest from the straight line connecting the outer corner and the inner corner of the eye. In the same manner, the face part detection unit 13 may calculate the degree of opening of the mouth as information indicating the degree of opening of the mouth.

[0019] In addition, the face part detection unit 13 may calculate the reliability of the eye opening degree, and output information indicating the reliability of the calculated eye opening degree by including it in the face part detection result. The reliability of the eye opening degree indicates whether the eye opening degree can be stably measured. When calculating the eye opening degree, for example, if the driver is wearing glasses, a landscape reflection image is reflected on the lens of the glasses, and the landscape reflection image interferes with the measurement. Therefore, if there is a landscape reflection, the reliability of the eye opening degree is set low. In order to detect such a landscape reflection, for example, the face part detection unit 13 is configured to acquire both eyes of the occupant from the face detection result, and if only one eye of the occupant can be acquired, it is determined that there is a landscape reflection. Since the occupant is generally seated on either the left or right door of the vehicle, landscape reflection often occurs only on one of the glasses of the glasses. Therefore, if only one eye of the occupant can be acquired, it can be estimated that a landscape reflection has occurred on the glass of the glasses in front of the other eye. Landscape reflection may be estimated by other methods, such as detecting the shape of a reflecting object reflected in the glasses. In addition to scenery reflection, the reliability of the eye opening degree is set low when the driver turns his / her face to the side (when the face direction yaw angle is large). The reliability of the eye opening degree may be calculated taking into consideration multiple indicators such as the presence or absence of scenery reflection and the face direction yaw angle.

[0020] (Feature Calculation Unit) The feature calculation unit 14 receives the face part detection result from the control unit, calculates a drowsiness feature that indicates a sign of drowsiness that appears in the face part detection result, and outputs the calculated drowsiness feature to the control unit. Examples of the drowsiness feature include the eye-closed time ratio that captures the increase in the driver's blinking due to drowsiness, and the number of yawns that capture yawning due to drowsiness. The eye-closed time ratio is calculated by calculating the ratio of the eye-closed time in a certain period of time. The eye-closed state means a state in which the distance between the upper eyelid and the lower eyelid is equal to or less than a predetermined threshold. The number of yawns is calculated by counting the number of yawns in a certain period of time. Yawning means a state in which the mouth is open by a predetermined threshold or more for a predetermined period of time or more. By determining whether the mouth is open by a predetermined threshold or more for a predetermined period of time or more, it is possible to distinguish between a case in which the mouth is open due to talking and a case in which the mouth is open due to yawning. The feature calculation unit 14 may calculate only one drowsiness feature amount, such as the proportion of time the eyes are closed, or may calculate a plurality of drowsiness feature amounts.

[0021] (Drowsiness score calculation section) The drowsiness score calculation unit 15 receives the value of the drowsiness feature from the control unit, calculates the drowsiness score, and outputs the calculated drowsiness score to the control unit. The drowsiness score is calculated using a trained machine learning model that has learned the relationship between the drowsiness feature and the drowsiness score using a machine learning model such as a support vector machine (SVM) or a random forest. That is, the drowsiness score calculation unit 15 inputs one or more feature amounts calculated by the feature calculation unit 14 to the trained model, and obtains the drowsiness score from the trained model, thereby calculating the drowsiness score. The drowsiness score may be output as a discrete value divided into multiple stages, or as a continuous value such as 0 to 1.

[0022] (Overdetection factor reliability calculation unit) The overdetection factor reliability calculation unit 16 receives the face part detection result from the control unit, determines the presence or absence of an overdetection factor based on the face part detection result, and when it is determined that an overdetection factor exists, further calculates the reliability of an overdetection factor index indicating the presence of the overdetection factor. The reliability includes, for example, a first reliability having a low reliability and a second reliability having a higher reliability than the first reliability, and the overdetection factor reliability calculation unit 16 calculates the first reliability or the second reliability as the reliability of the overdetection factor index. The reliability may be further subdivided. The overdetection factor reliability calculation unit 16 outputs to the control unit an overdetection factor index indicating the presence of an overdetection factor and the reliability of the overdetection factor index indicating the likelihood that an overdetection factor exists.

[0023] Here, the overdetection factor index means an index indicating whether a factor that may cause overdetection has occurred. Overdetection means that the driver is erroneously determined to be drowsy even though the driver is not actually feeling drowsy. Overdetection is a concept that is distinguished from non-detection, and non-detection means that the driver is erroneously determined to be not drowsy even though the driver is actually feeling drowsy. Examples of such overdetection factor indexes include a downward gaze index that indicates that the driver's gaze is directed downward, and a conversation index that indicates that the driver is talking to a passenger.

[0024] In the case of the downward gaze index, for example, whether the eye is closed or downward gaze can be determined by distinguishing between closed and downward gaze based on the distance between the upper and lower eyelids. In other words, since the distance between the upper and lower eyelids is generally longer when the eye is downward gazed than when the eye is closed, it is possible to distinguish between closed and downward gaze based on the distance between the upper and lower eyelids.

[0025] In the case of an example of a conversation index, an open mouth where the upper and lower lips are equal to or greater than a predetermined threshold and a closed mouth where the lips are less than the threshold are detected, and the speed of the opening and closing of the mouth is calculated. By distinguishing between yawning and talking based on the speed of the opening and closing of the mouth, it is possible to determine whether or not the action is talking. In other words, since the speed of the opening and closing of the mouth is generally faster in talking than in yawning, it is possible to distinguish between yawning and talking based on the speed of the opening and closing of the mouth.

[0026] The reliability of the overdetection factor index is a value that indicates the likelihood of the overdetection factor index, that is, the likelihood that an overdetection factor exists. In the case of the example of the downward gaze index, the reliability of the downward gaze index decreases when there is a possibility that the downward gaze index cannot be correctly determined. For example, the time series variance of the coordinates of the upper and lower eyelids is calculated, and if the value is equal to or greater than a threshold, it is determined that the coordinates of the upper and lower eyelids cannot be stably detected, and the reliability is determined to be low. Conversely, if the value is less than the threshold, the reliability is determined to be high. In addition, the reliability of the downward gaze index may be determined to be high or low using the reliability of the eye opening degree received as the face detection result and a preset threshold. In the example of the conversation index, if the time series variance of the speed of the movement due to opening and closing the mouth is equal to or greater than a preset threshold, the reliability may be determined to be low, and if the value is less than the threshold, the reliability may be determined to be high.

[0027] (State transition determination unit) The state transition determination unit 17 receives the drowsiness score, the overdetection factor index, and the reliability of the overdetection factor index from the control unit, determines the drowsiness state of the driver, and outputs the determined drowsiness state. If the overdetection factor index and the reliability of the overdetection factor index do not exist for the current frame, they are not output, and in such a case the state transition determination unit 17 does not receive either the overdetection factor index or the reliability of the overdetection factor index.

[0028] As an example, the state transition determination unit 17 maintains the current drowsiness state if the drowsiness score is within a threshold range set for the current drowsiness state based on the drowsiness score and a predetermined state transition threshold. Also, if the drowsiness score is smaller than the drowsiness descent threshold, the state transition determination unit 17 transitions to a weaker drowsiness state and outputs the drowsiness state after the transition. The drowsiness descent threshold is a threshold for determining a transition from the current drowsiness state to a weaker drowsiness state.

[0029] Furthermore, when the drowsiness score is equal to or greater than the drowsiness increase threshold, the state transition determination unit 17 may check the presence or absence of an overdetection factor index, and if the check result shows that there is no overdetection factor, the state transition determination unit 17 may transition the drowsiness state according to the drowsiness score. The drowsiness increase threshold is a threshold for determining a transition from the current drowsiness state to a stronger drowsiness state. When the check result shows that there is an overdetection factor and the reliability of the overdetection factor index is high, the state transition determination unit 17 may determine that the increase in the drowsiness score is not due to an increase in drowsiness but due to an overdetection factor, and may not transition the drowsiness state. That is, the current drowsiness state may be maintained. When the check result shows that there is an overdetection factor and the reliability of the overdetection factor index is low, the state transition determination unit 17 may perform a correction so that the transition of the drowsiness state becomes more difficult. For example, the correction may be performed by increasing the threshold for transitioning from a first drowsiness state having a low degree of drowsiness to a second drowsiness state having a higher degree of drowsiness than the first drowsiness state. As another example, the correction may be performed by transitioning to an intermediate state between the first drowsiness state and the second drowsiness state.

[0030] Next, a hardware configuration example of the drowsiness estimation device 100 will be described with reference to Fig. 2A and Fig. 2B. Each functional unit of the drowsiness estimation device 100 is realized by a processing circuit. The processing circuitry may be a dedicated processing circuit 100a as shown in Fig. 2A, or a processor 100b that executes a program stored in a memory 100c as shown in Fig. 2B.

[0031] When the processing circuitry is a dedicated processing circuit 100a, the dedicated processing circuit 100a may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of these. Each functional unit may be realized by a plurality of separate processing circuits, or each functional unit may be realized by a single processing circuit.

[0032] When the processing circuitry is the processor 100b, each functional unit is realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 100c. The processor 100b realizes each functional unit by reading and executing the programs stored in the memory. Examples of the memory 100c include non-volatile or volatile semiconductor memories such as random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM), as well as magnetic disks, flexible disks, optical disks, compact disks, mini disks, and DVDs.

[0033] In addition, some of the multiple functional units may be realized by dedicated hardware, and the other parts may be realized by software or firmware. In this way, the processing circuit can realize each functional unit by hardware, software, firmware, or a combination of these.

[0034] <Operation> Next, a schematic operation of the drowsiness estimation device 100 according to the first embodiment of the present disclosure will be described with reference to the flowchart of FIG.

[0035] In step ST11, the drowsiness score calculation unit 15 calculates a drowsiness score indicating the degree of drowsiness of the driver based on an image of the driver's face using a trained machine learning model. As an example, the drowsiness score calculation unit 15 uses a trained machine learning model that has learned the relationship between the drowsiness feature amount extracted from the image of the driver's face and the drowsiness score, and inputs the drowsiness feature amount extracted from the driver's face into the machine learning model to obtain the driver's drowsiness score.

[0036] In step ST12, the overdetection factor reliability calculation unit 16 determines whether there is an overdetection factor that affects the calculation of the drowsiness score based on an image of the driver's face, and calculates the reliability of the overdetection factor index when it is determined that there is an overdetection factor. These determination and calculation processes are performed on a rule basis. As an example, the overdetection factor reliability calculation unit 16 determines whether there is an overdetection factor based on the driver's face part detection result, and calculates the reliability of the overdetection factor index when there is an overdetection factor. For example, it determines whether there is downward gaze based on the distance between the upper and lower eyelids, and when there is downward gaze, calculates the reliability of the downward gaze index from the time series variance of the coordinates of the upper and lower eyelids.

[0037] Note that step ST11 and step ST12 may be performed in any order, or may be performed simultaneously.

[0038] In step ST13, the state transition determination unit 17 determines the drowsiness state of the driver based on the calculated drowsiness score, the calculated overdetection factor index, and the reliability of the calculated overdetection factor index.

[0039] Next, the operation of the overdetection factor reliability calculation unit 16 will be described in more detail with reference to the flowchart of Fig. 4. The flowchart of Fig. 4 describes the case where the overdetection factor is downward gaze, but the process may be performed in the same manner as in Fig. 4 when the overdetection factor is other than conversation. It is assumed at the start of the flowchart of Fig. 4 that the overdetection factor reliability calculation unit 16 has received information on the eye opening / closing state, such as the degree of eye opening, as a face part detection result.

[0040] In short, the overdetection factor reliability calculation unit 16 analyzes the eye condition and judges the presence or absence of downgaze according to the multiple eye conditions. When the overdetection factor reliability calculation unit 16 judges the presence of downgaze, it further judges the reliability of the judgment that downgaze is present. This will be explained in more detail below.

[0041] In step ST101, it is confirmed whether the eyes are closed. The confirmation of whether the eyes are closed or not can be performed, for example, by determining whether the degree of eye opening received as the face part detection result is equal to or less than a predetermined threshold. If the result is "NO", it is determined in step ST111 that there is no downward gaze. That is, if the eyes are not closed, in other words, if the eyes are open, the process proceeds to step ST111, and it is determined in step ST111 that there is no downward gaze. On the other hand, if the result is "YES" in step ST101, that is, if it is determined that the eyes are closed, the process proceeds to steps ST102 to ST104, and the rate of both eyelids open is calculated to determine whether the eyes are really closed or are looking downward. Note that, if the overdetection factor is conversation, it may be determined that the eyes are open instead of closed.

[0042] In step ST102, curve fitting is performed on the upper and lower eyelids. This will be described with reference to Fig. 5. Fig. 5 shows the state in which curve fitting is performed on one of a person's eyes. As shown in Fig. 5, both the upper eyelid curve and the lower eyelid curve connecting the coordinates of the inner corner and the outer corner of the eye are calculated. In addition, the vertex coordinates of the upper eyelid are calculated from the calculated upper eyelid curve, and the vertex coordinates of the lower eyelid are calculated from the calculated lower eyelid curve.

[0043] In step ST103, the inter-eyelid distance shown in Fig. 5 is measured based on the vertex coordinates of the upper eyelid and the lower eyelid calculated in step ST102. The inter-eyelid distance is the difference between the vertex coordinates of the upper eyelid and the vertex coordinates of the lower eyelid.

[0044] In step ST104, the eyelid opening rate is calculated. The eyelid opening rate is calculated according to (eyelid opening rate)=(eyelid distance) / (eyelid distance when closed). Here, "when closed" in the denominator means a case where the eyes are determined to be closed using the above-mentioned eye opening degree. "Eyelid distance when closed" is the distance between both eyelids in such a case, and is a reference value measured for each driver. This reference value is calculated by averaging the eyelid distance when the eyes are closed multiple times over a certain period of time. This calculation may be performed by instructing the driver to close the eyes before the vehicle starts to travel and measuring the eyes closed, or by measuring the eyes closed of the driver immediately after the vehicle starts to travel. If the "eyelid distance when closed" in the denominator is calculated to be 0, the measurement may be performed again. Alternatively, to prevent the denominator from becoming 0, a small value Δd can be added to the "distance between the eyelids when the eyes are closed" and the rate at which both eyelids are open can be calculated as (rate at which both eyelids are open) = (distance between the eyelids) / (distance between the eyelids when the eyes are closed + Δd).

[0045] In step ST106, it is determined whether the eyelid opening rate is equal to or greater than a preset threshold. In this way, for the eye state determined to be closed, a true eye closed state is distinguished from a state determined to be closed despite the eye being looked down. Note that, if the overdetection factor is conversation, instead of the eyelid opening rate, it may be determined whether the speed of the opening and closing movements is equal to or greater than a threshold.

[0046] If the answer is "NO" in step ST106, that is, if the eyelid opening rate is less than the threshold, it is considered that the driver really has his eyes closed. In such a case, it is determined in step ST111 that the driver is not looking down.

[0047] If the answer is "YES" in step ST106, that is, if the eyelid opening rate is equal to or greater than the threshold, it is considered that the driver is not closing his / her eyes but looking down. Therefore, in such a case, it is determined that the driver is looking down (steps ST109 and ST110). That is, a flag for a downward gaze indicator is set.

[0048] When determining that there is a downgaze, the overdetection factor reliability calculation unit 16 may further determine the reliability of the determination that there is a downgaze, that is, the reliability of the downgaze index (steps ST107 to ST108).

[0049] In step ST107, the time series variance of the coordinates of the upper and lower eyelids is calculated to determine the reliability of the downward gaze index. If the time series variance is equal to or greater than a predetermined threshold (if "YES" in step ST108), it is considered that the result of the upper and lower eyelid curve fitting in step ST102 is unstable due to some factor, and therefore, in step ST109, it is determined that there is downward gaze (low reliability). In other words, it is determined that there is downward gaze as a cause of overdetection, but the reliability is low. On the other hand, if the time series variance is less than a predetermined threshold (if "NO" in step ST108), it is determined that there is downward gaze (high reliability) in step ST110. In other words, it is determined that there is downward gaze as a cause of overdetection, and the reliability is high. Note that, when the cause of overdetection is conversation, instead of the time series variance of the coordinates of the upper and lower eyelids, the time series variance of the speed of the movement due to opening and closing the mouth may be calculated, and it may be determined whether the time series variance of the speed of this movement is equal to or greater than a threshold.

[0050] Here, the variance of the coordinates of the upper and lower eyelids is used as the reliability, but the time series average value of the reliability of the eye opening degree received as the face detection result may be taken as the reliability of the downward gaze index. For example, if the reliability of the eye opening degree is high (=1) at time t, high (=1) at time t+1, low (=0) at time t+2, and high (=1) at time t+3, the time series average value of these is 3 / 4=0.75. Since this value is greater than 0.5, which is the intermediate value between the high and low reliability of the eye opening degree, the reliability of the downward gaze index may be determined to be high. As another example, if the reliability of the eye opening degree is low (=0) at time t, high (=1) at time t+1, low (=0) at time t+2, and low (=0) at time t+3, the time series average value of these is 1 / 4=0.25. Since this value is smaller than 0.5, which is the intermediate value of the reliability of the eye opening degree, the reliability of the downward gaze index may be determined to be low.

[0051] Next, the operation of state transition determination unit 17 will be described in more detail with reference to the flowchart of FIG.

[0052] In step ST113, it is determined whether the drowsiness score, which is the output of the drowsiness score calculation unit 15, is equal to or greater than the drowsiness rising threshold for transitioning from the current drowsiness state to a stronger drowsiness state. The drowsiness state is divided into a plurality of stages, and a threshold for the drowsiness score for transition between each state is previously determined. With reference to FIG. 7 used in the description of the modified example described later, if the current drowsiness state is the "drowsiness state 2", it is determined whether the drowsiness score p is equal to or greater than the drowsiness rising threshold T23 for transitioning from the "drowsiness state 2" to the stronger drowsiness state "drowsiness state 3". If the current drowsiness state is the "drowsiness state 1", it is determined whether the drowsiness score p is equal to or greater than the drowsiness rising threshold T12 for transitioning from the "drowsiness state 1" to the stronger drowsiness state "drowsiness state 2". If the determination result in step ST113 is "NO", the process proceeds to step ST118, and the state transition is determined based on the drowsiness score. On the other hand, if the determination result is "YES", the process proceeds to step ST114.

[0053] In step ST114, the overdetection factor index, which is the output of the overdetection factor reliability calculation unit 16, is checked to determine whether there is an overdetection factor. If "NO", the process proceeds to step ST118, where a state transition is determined based on the drowsiness score. If "YES", the process proceeds to step ST115.

[0054] In step ST115, the reliability output from the overdetection factor reliability calculation unit 16 is checked to determine whether the reliability is high. If the reliability is "YES", it is determined in step ST116 that there is no state transition. That is, if the reliability of the overdetection factor is high, for example, if the reliability of downward gaze is high, no transition between multiple drowsiness states, which is an evaluation of the driver's drowsiness state, is made, and the determination of the drowsiness state level made for the immediately preceding frame is maintained.

[0055] On the other hand, if the answer is "NO," a correction is performed in step ST117 so that the drowsiness state is less likely to increase. The correction method may be a method of adjusting a transition threshold for the drowsiness score, or a method of transitioning to a state that is an average of a destination drowsiness state based on the drowsiness score and the current drowsiness state.

[0056] As described above, according to the first embodiment of the present disclosure, the drowsiness score calculated using the machine learning model based on the change in the feature amount is corrected by the overdetection factor index determined by the rule-based method and its reliability. Therefore, even if it is difficult to distinguish between cases where drowsiness is present and cases where drowsiness is not present using only the machine learning model, it is possible to distinguish between these cases using the rule-based method.

[0057] Moreover, according to the first embodiment, since the reliability of the overdetection factor index is used, the drowsiness level can be estimated more accurately than when the overdetection factor index is simply used.

[0058] Variations. The drowsiness estimation device 100 according to the first embodiment judges the transition of the drowsiness state using the drowsiness score based on the machine learning model, the rule-based overdetection factor index and its reliability. When judging the transition of the drowsiness state by such a method, the drowsiness state may rise based on the judgment result for a certain frame, and the drowsiness state may immediately fall based on the judgment result for the next frame. However, it is known that the actual drowsiness does not fall immediately after rising.

[0059] In consideration of such actual drowsiness transitions, the transition thresholds between different levels of drowsiness states may be set to different values. Specifically, the transition threshold for determining a transition from a first drowsiness state having a stronger drowsiness level to a second drowsiness state having a weaker drowsiness level may be set lower than the transition threshold for determining a transition from the second drowsiness state to the first drowsiness state.

[0060] A more detailed description will be given with reference to FIG. 7. FIG. 7 shows three stages of drowsiness states with different drowsiness levels. The relationship between these three drowsiness states is as follows: "drowsiness state 1" is the drowsiness state with the weakest drowsiness level, "drowsiness state 3" is the drowsiness state with the strongest drowsiness level, and "drowsiness state 2" is the drowsiness state with an intermediate drowsiness level between the drowsiness state 1 and the drowsiness state 2. In this case, the drowsiness descent threshold T21 for determining the transition from the drowsiness state 2 to the drowsiness state 1 is set to be α lower than the drowsiness rise threshold T12 for determining the transition from the drowsiness state 1 to the drowsiness state 2. Similarly, the drowsiness descent threshold T32 from the drowsiness state 3 to the drowsiness state 2 is set to be β lower than the drowsiness rise threshold T23 from the drowsiness state 2 to the drowsiness state 3.

[0061] The state transition determination unit 17 determines the transition of the drowsiness state based on the state transition thresholds set as described above. That is, in step ST118 of Fig. 6, the state transition determination unit 17 determines the transition of the drowsiness state using the drowsiness score received from the drowsiness score calculation unit 15, the drowsiness increasing threshold, and the drowsiness decreasing threshold.

[0062] According to the above modification, it is possible to provide hysteresis (α, β) to the transition between drowsiness states. Therefore, it is possible to perform a drowsiness determination in accordance with the actual change in drowsiness, and it is possible to prevent erroneous estimation of the drowsiness state.

[0063] <Additional Notes> Certain aspects of the various embodiments described above are summarized below.

[0064] (Appendix 1) The drowsiness estimation device (100) of Appendix 1 includes a drowsiness score calculation unit (15) that calculates a drowsiness score indicating the degree of drowsiness of a vehicle occupant using a trained machine learning model based on an image of the face of the vehicle occupant, an overdetection factor reliability calculation unit (16) that determines whether or not there is an overdetection factor that affects the calculation of the drowsiness score based on the image, and when it is determined that the overdetection factor exists, calculates an overdetection factor index indicating the existence of the overdetection factor and a reliability of the overdetection factor index in a rule-based manner, and a state transition determination unit (17) that determines which of a plurality of drowsiness states the occupant is in, including a first drowsiness state having a lower degree of drowsiness and a second drowsiness state having a higher degree of drowsiness than the first drowsiness state, based on the calculated drowsiness score, the calculated overdetection factor index, and the reliability of the calculated overdetection factor index.

[0065] (Appendix 2) The drowsiness estimation device of Appendix 2 is the drowsiness estimation device described in Appendix 1, wherein the reliability of the overdetection factor index includes a first reliability having a low reliability and a second reliability having a higher reliability than the first reliability, and when the calculated overdetection factor index is present and the calculated reliability is the second reliability, the state transition determination unit maintains the determination of the immediately preceding drowsiness state without transitioning the determination regarding the drowsiness state between the multiple drowsiness states.

[0066] (Appendix 3) The drowsiness estimation device of Appendix 3 is the drowsiness estimation device described in Appendix 1 or 2, wherein when the calculated overdetection factor index exists and the calculated reliability is the first reliability, the state transition determination unit performs a correction so as to make it more difficult to transition from the first drowsiness state to the second drowsiness state.

[0067] (Appendix 4) The drowsiness estimation device of Appendix 4 is the drowsiness estimation device described in Appendix 3, wherein when the calculated overdetection factor index exists and the calculated reliability is the first reliability, the state transition determination unit performs the correction by increasing a threshold for transitioning from the first drowsiness state to the second drowsiness state.

[0068] (Appendix 5) The drowsiness estimation device of Appendix 5 is the drowsiness estimation device described in Appendix 3, wherein when the calculated overdetection factor index exists and the calculated reliability is the first reliability, the state transition determination unit performs the correction by transitioning to an intermediate state between the first drowsiness state and the second drowsiness state.

[0069] (Appendix 6) The drowsiness estimation device of Appendix 6 is the drowsiness estimation device described in Appendix 2, wherein the overdetection factor is the occupant looking downward, and the overdetection factor reliability calculation unit calculates the reliability of the overdetection factor index when an eyelid opening rate between both eyelids of the occupant is equal to or greater than a predetermined threshold.

[0070] (Appendix 7) The drowsiness estimation device of Supplementary Note 7 is the drowsiness estimation device described in Supplementary Note 6, wherein the overdetection factor reliability calculation unit determines that the reliability of the overdetection factor index is the first reliability when a time series variance of the vertex coordinates of the upper and lower eyelids is equal to or greater than a predetermined threshold, and determines that the reliability of the overdetection factor index is the second reliability when the time series variance of the vertex coordinates of the upper and lower eyelids is less than the predetermined threshold.

[0071] (Appendix 8) The drowsiness estimation device of Supplementary note 8 is the drowsiness estimation device described in Supplementary note 6, wherein the over-detection factor reliability calculation unit calculates the reliability of the over-detection factor index based on the reliability of an eye openness degree.

[0072] (Appendix 9) The drowsiness estimation device of Appendix 9 is the drowsiness estimation device described in Appendix 2, wherein the overdetection factor is conversation of the occupant, and the overdetection factor reliability calculation unit calculates the reliability of the overdetection factor index when the speed of the occupant's mouth opening and closing movement is equal to or greater than a predetermined threshold.

[0073] (Appendix 10) The drowsiness estimation device of Supplementary Note 10 is the drowsiness estimation device described in Supplementary Note 9, wherein the overdetection factor reliability calculation unit determines that the reliability of the overdetection factor index is the first reliability when a time series variance of the vertex coordinates of the upper and lower lips is equal to or greater than a predetermined threshold, and determines that the reliability of the overdetection factor index is the second reliability when the time series variance of the vertex coordinates of the upper and lower lips is less than the predetermined threshold.

[0074] (Appendix 11) The drowsiness estimation device of Appendix 11 is a drowsiness estimation device described in any one of Appendixes 1 to 10, further comprising a feature calculation unit (14) that calculates a drowsiness feature indicating a sign of drowsiness based on the image, wherein the trained machine learning model is a trained model that has learned the relationship between the drowsiness feature and a drowsiness score, and the drowsiness score calculation unit calculates the drowsiness score of the occupant from the calculated drowsiness feature using the trained machine learning model.

[0075] (Appendix 12) The drowsiness estimation device of Appendix 12 is a drowsiness estimation device described in any one of Appendixes 1 to 11, wherein a threshold value for determining a transition from the first drowsiness state to the second drowsiness state is set higher than a threshold value for determining a transition from the second drowsiness state to the first drowsiness state.

[0076] (Appendix 13) The drowsiness estimation method of Appendix 13 is a drowsiness estimation method performed by a drowsiness estimation device including a drowsiness score calculation unit (15), an overdetection factor reliability calculation unit (16), and a state transition determination unit (17), and includes a step (ST11) in which the drowsiness score calculation unit calculates a drowsiness score indicating the degree of drowsiness of the vehicle occupant using a trained machine learning model based on an image of the face of the vehicle occupant, a step (ST12) in which the overdetection factor reliability calculation unit determines, based on the image, whether there is an overdetection factor that affects the calculation of the drowsiness score, and if it is determined that the overdetection factor exists, calculates, on a rule-based basis, a reliability of an overdetection factor index that indicates the likelihood that the overdetection factor exists, and a step (ST13) in which the state transition determination unit determines the drowsiness state of the occupant based on the calculated drowsiness score and the calculated reliability.

[0077] It is possible to combine the embodiments, and to modify or omit each embodiment as appropriate. [Industrial Applicability]

[0078] The drowsiness estimation technology of the present disclosure can be used as a technology for realizing a drowsiness estimation function for PMS. [Explanation of symbols]

[0079] 1 vehicle, 2 imaging device, 10 face information detection unit, 11 image acquisition unit, 12 face detection unit, 13 face part detection unit, 14 feature amount calculation unit, 15 drowsiness score calculation unit, 16 overdetection factor reliability calculation unit, 17 state transition determination unit, 20 drowsiness determination unit, 100 drowsiness estimation device, 100a processing circuit, 100b processor, 100c memory.

Claims

1. a drowsiness score calculation unit that calculates a drowsiness score indicating a degree of drowsiness of a vehicle occupant by using a trained machine learning model based on an image of a face of the vehicle occupant; an overdetection factor reliability calculation unit that determines whether or not there is an overdetection factor that affects the calculation of the drowsiness score based on the image, and calculates, when it is determined that there is an overdetection factor, an overdetection factor index indicating the presence of the overdetection factor and a reliability of the overdetection factor index in a rule-based manner; a state transition determination unit that determines which of a plurality of drowsiness states the occupant is in, including a first drowsiness state having a low degree of drowsiness and a second drowsiness state having a higher degree of drowsiness than the first drowsiness state, based on the calculated drowsiness score, the calculated overdetection factor index, and a reliability of the calculated overdetection factor index; and Equipped with The reliability of the overdetection factor index includes a first reliability having a low reliability and a second reliability having a higher reliability than the first reliability, When the calculated overdetection factor index exists and the calculated reliability is the second reliability, the state transition determination unit maintains the determination of the immediately preceding drowsiness state without transitioning the determination of the drowsiness state between the plurality of drowsiness states. Drowsiness estimation device.

2. When the calculated overdetection factor index exists and the calculated reliability is the first reliability, the state transition determination unit performs a correction so that a transition from the first drowsiness state to the second drowsiness state becomes more difficult. The drowsiness estimation device according to claim 1 .

3. when the calculated overdetection factor index exists and the calculated reliability is the first reliability, the state transition determination unit performs the correction by increasing a threshold for transitioning from the first drowsiness state to the second drowsiness state. The drowsiness estimation device according to claim 2.

4. the state transition determination unit, when the calculated overdetection factor index exists and the calculated reliability is the first reliability, performs the correction by transitioning to an intermediate state between the first drowsiness state and the second drowsiness state. The drowsiness estimation device according to claim 2.

5. the overdetection cause is the occupant looking downward, the overdetection factor reliability calculation unit calculates the reliability of the overdetection factor index when an eyelid opening rate between both eyelids of the occupant is equal to or greater than a predetermined threshold value. The drowsiness estimation device according to claim 1 .

6. the overdetection factor reliability calculation unit determines that the reliability of the overdetection factor index is the first reliability when a time series variance of the vertex coordinates of the upper and lower eyelids is equal to or greater than a predetermined threshold, and determines that the reliability of the overdetection factor index is the second reliability when the time series variance of the vertex coordinates of the upper and lower eyelids is less than the predetermined threshold. A drowsiness estimation device according to claim 5.

7. the overdetection factor reliability calculation unit calculates the reliability of the overdetection factor index based on the reliability of the eye openness degree; A drowsiness estimation device according to claim 5.

8. the overdetection cause is a conversation between the occupants, the overdetection factor reliability calculation unit calculates the reliability of the overdetection factor index when a speed of the occupant's mouth opening and closing movement is equal to or greater than a predetermined threshold. The drowsiness estimation device according to claim 1 .

9. the overdetection factor reliability calculation unit determines that the reliability of the overdetection factor index is the first reliability when a time series variance of the vertex coordinates of the upper and lower lips of the occupant is equal to or greater than a predetermined threshold, and determines that the reliability of the overdetection factor index is the second reliability when the time series variance of the vertex coordinates of the upper and lower lips is less than the predetermined threshold. The drowsiness estimation device according to claim 8.

10. A feature amount calculation unit is further provided to calculate a drowsiness feature amount indicating a sign of drowsiness based on the image, The trained machine learning model is a trained model that has learned the relationship between a drowsiness feature amount and a drowsiness score, The drowsiness score calculation unit calculates a drowsiness score of the occupant from the calculated drowsiness feature amount by using the trained machine learning model. The drowsiness estimation device according to claim 1 .

11. a threshold value for determining a transition from the first drowsiness state to the second drowsiness state is set higher than a threshold value for determining a transition from the second drowsiness state to the first drowsiness state. The drowsiness estimation device according to any one of claims 1 to 10.

12. a drowsiness score calculation unit that calculates a drowsiness score indicating a degree of drowsiness of a vehicle occupant by using a trained machine learning model based on an image of a face of the vehicle occupant; an overdetection factor reliability calculation unit that determines whether or not there is an overdetection factor that affects the calculation of the drowsiness score based on the image, and calculates, when it is determined that there is an overdetection factor, an overdetection factor index indicating the presence of the overdetection factor and a reliability of the overdetection factor index in a rule-based manner; a state transition determination unit that determines which of a plurality of drowsiness states the occupant is in, including a first drowsiness state having a low degree of drowsiness and a second drowsiness state having a higher degree of drowsiness than the first drowsiness state, based on the calculated drowsiness score, the calculated overdetection factor index, and a reliability of the calculated overdetection factor index; and Equipped with a threshold value for determining a transition from the first drowsiness state to the second drowsiness state is set higher than a threshold value for determining a transition from the second drowsiness state to the first drowsiness state. Drowsiness estimation device.

13. A drowsiness estimation method performed by a drowsiness estimation device including a drowsiness score calculation unit, an overdetection factor reliability calculation unit, and a state transition determination unit, a step of the drowsiness score calculation unit calculating a drowsiness score indicating a degree of drowsiness of the occupant by using a trained machine learning model based on an image of a face of the occupant of the vehicle; the overdetection factor reliability calculation unit determines whether or not there is an overdetection factor that affects the calculation of the drowsiness score based on the image, and when it is determined that there is an overdetection factor, calculates a reliability of an overdetection factor index that indicates a likelihood that the overdetection factor exists on a rule basis; a step of the state transition determination unit determining which of a plurality of drowsiness states the occupant is in, including a first drowsiness state having a low degree of drowsiness and a second drowsiness state having a higher degree of drowsiness than the first drowsiness state, based on the calculated drowsiness score, the calculated overdetection factor index, and the calculated reliability; Equipped with The reliability of the overdetection factor index includes a first reliability having a low reliability and a second reliability having a higher reliability than the first reliability, a step of the state transition determination unit, when the calculated overdetection factor index exists and the calculated reliability is the second reliability, not transitioning the determination regarding the drowsiness state between the plurality of drowsiness states and maintaining the determination regarding the immediately preceding drowsiness state; The drowsiness estimation method includes:

Images