Subject status determination system, subject status determination method, and program

Abstract

This invention provides a subject state determination system, a subject state determination method, and a program that can accurately estimate the subject's state from the actions expressed on the subject's face. [Solution] The subject state determination system includes: an action detection unit that detects the subject's state from actions expressed on the subject's face; an estimation unit that estimates the degree of the subject's state from actions expressed on the subject's face; and a determination unit that determines whether or not the subject is in a specific state based on either or both of the detection result of the action detection unit and the estimation result of the estimation unit.

Description

Technical Field

[0001] The present invention relates to a subject state determination system, a subject state determination method, and a program.

Background Art

[0002] Conventionally, a technique for imaging an image including a subject's face and determining the subject's state is known. As a document in which this type of technique is disclosed, for example, Patent Document 1 is known. In Patent Document 1, as an example of means for determining a driver's drowsiness, an image of the driver is captured using a camera, and the driver's blinking, lip movement, etc. are detected from the captured image, or the driver's stretching of the body, twisting of the body, etc. are detected. A drowsiness determination device is disclosed. When the drowsiness determination device determines that the driver is in a drowsy state, it outputs a notification to inform the driver that he / she is in a drowsy state.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Since the driver imaged by the camera frowns his / her face in the sunlight or moves his / her mouth during eating or drinking, the facial expression changes due to various factors. Therefore, the driver's blinking, lip movement, etc. do not necessarily indicate drowsiness. If the driver is misjudged as being drowsy when he / she is actually not drowsy, he / she will feel bothered by the notification that he / she is in a drowsy state.

[0005] The present invention has been made in view of such a situation, and an object thereof is to provide a subject state determination system, a subject state determination method, and a program that can accurately estimate the state of a subject from the actions expressed on the subject's face.

Means for Solving the Problems

[0006] To achieve the above objective, one aspect of the present invention is a subject state determination system comprising: an action detection unit that detects the state of a subject from actions expressed on the subject's face; an estimation unit that estimates the degree of the subject's state from actions expressed on the subject's face; and a determination unit that determines whether or not the subject is in a specific state based on either or both of the detection result of the action detection unit and the estimation result of the estimation unit.

[0007] Furthermore, one aspect of the present invention is a method for determining the state of a subject, which is performed by a computer that determines the state of a subject, and comprises: an action detection step for detecting the state of the subject from actions expressed on the subject's face; an estimation step for estimating the degree of the subject's state from actions expressed on the subject's face; and a determination step for determining whether or not the subject is in a specific state based on either or both of the detection result of the action detection step and the estimation result of the estimation step.

[0008] Furthermore, one aspect of the present invention is a program for causing a computer to perform the following steps: an action detection step for detecting the state of a subject from actions expressed on the subject's face; an estimation step for estimating the degree of the subject's state from actions expressed on the subject's face; and a determination step for determining whether or not the subject is in a specific state based on either or both of the detection result of the action detection step and the estimation result of the estimation step. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a subject state determination system, a subject state determination method, and a program that can accurately estimate the state of a subject from the actions expressed on the subject's face. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram showing a target person status determination system according to one embodiment of the present invention. [Figure 2]This is a block diagram showing the hardware configuration of the subject status determination device according to this embodiment. [Figure 3] This is a functional block diagram showing an example of the functional configuration of the subject status determination system of this embodiment. [Figure 4] This diagram illustrates the normalization and frontalization processes performed by the preprocessing unit. [Figure 5] This diagram illustrates the process of detecting yawns in a subject. [Figure 6] This diagram illustrates the process of detecting a subject's slow blinking. [Figure 7] This table shows an example of input data used in a sleepiness estimation model. [Figure 8] This graph illustrates the method for calculating the first feature as input data. [Figure 9] This graph illustrates the method for calculating the second feature used as input data. [Figure 10] This graph explains how to calculate the third feature used as input data. [Figure 11] This graph explains how to calculate the fourth feature as input data. [Figure 12] This graph explains how to calculate the fifth feature as input data. [Figure 13] This flowchart shows an example of the processing flow of the subject status determination system of this embodiment. [Modes for carrying out the invention]

[0011] The following describes one embodiment of the present invention with reference to the drawings. Figure 1 is a schematic diagram showing a subject status determination system 100 according to one embodiment of the present invention.

[0012] <System Configuration> As shown in FIG. 1, the subject state determination system 100 includes an imaging device 10 that captures an image including the face of a subject, a subject state determination device 1 that determines the state of the subject from the image captured by the imaging device 10, and a notification device 2. In the present embodiment, the subject whose state is determined by the subject state determination device 1 will be described, for example, in the case of the driver of the vehicle 101.

[0013] The imaging device 10 is installed at a position where it can capture the eyes and mouth of the face of the driver driving the vehicle 101. The imaging device 10 captures a moving image composed of temporally continuous images in order to detect the expression of the driver that changes over time. The imaging device 10 has a communication function for communicating with the subject state determination device 1 via a predetermined communication network, and transmits the captured image to the subject state determination device 1 by the communication function. The communication function may be possessed by the imaging device 10 itself, or may be realized by a communication device arranged outside the imaging device 10.

[0014] The subject state determination device 1 analyzes the image acquired by the imaging device 10 and determines the state of the driver. The subject state determination device 1 of the present embodiment captures the face of the driver of the vehicle 101 and determines the degree of drowsiness of the driver. The subject state determination device 1 is arranged outside the vehicle 101 and acquires the image captured by the imaging device 10 via a communication network such as the Internet. Alternatively, the subject state determination device 1 may be arranged inside the vehicle 101 and configured to acquire an image from the imaging device 10 via wireless communication, a communication cable, or the like.

[0015] The notification device 2 is a device that notifies the driver of the state of the driver based on the determination result of the subject state determination device 1. The notification device 2 is composed of, for example, a display, a speaker, or a combination thereof. The notification device 2 of the present embodiment alerts the driver who is determined to be drowsy by the subject state determination device 1 by voice or image.

[0016] <Hardware Configuration> Next, an example of the hardware configuration of the subject status determination device 1 will be described. Figure 2 is a block diagram showing the hardware configuration of the subject status determination device 1 in this embodiment. The subject status determination device 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20.

[0017] The CPU 11 executes various processes according to the program stored in the ROM 12 or the program loaded from the storage unit 18 into the RAM 13. The RAM 13 also stores data necessary for the CPU 11 to execute various processes. The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface 15 is also connected to this bus 14.

[0018] The input / output interface 15 is connected to an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20. The output unit 16 consists of a display, speaker, etc., and outputs various information as images and sounds. The input unit 17 consists of a keyboard, mouse, etc., and inputs various information. The storage unit 18 consists of a hard disk, DRAM (Dynamic Random Access Memory), etc., and stores various data. The communication unit 19 communicates with the subject status determination device 1 via a communication network including the Internet.

[0019] The drive 20 is appropriately equipped with removable media 21, which may consist of a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory. Programs read from the removable media 21 by the drive 20 are installed in the storage unit 18 as needed. The removable media 21 can also store various types of data stored in the storage unit 18, just like the storage unit 18.

[0020] Note that the hardware configuration described with reference to Figure 2 is merely an example; any computer capable of implementing the function to determine the subject's condition, as described later, will suffice.

[0021] <Functional Configuration> Next, the functional configuration of the subject status determination system 100 will be described with reference to Figure 3. Figure 3 is a functional block diagram showing an example of the functional configuration of the subject status determination system 100 in this embodiment.

[0022] As shown in Figure 3, the subject state determination device 1 includes a pre-processing unit 31, a drowsiness behavior detection unit 32, a drowsiness estimation unit 33, and a drowsiness determination unit 34 as functional units implemented on the processor (CPU 11).

[0023] The preprocessing unit 31 will be described with reference to Figure 4. Figure 4 is a diagram illustrating the normalization and frontalization processing performed by the preprocessing unit 31. The preprocessing unit 31 performs normalization and frontalization processing on landmark points that represent facial features such as eyes and noses extracted from the image, in order to unify the size and orientation of the face. For example, the extraction of landmark points from the image can be done using the publicly known Facemesh library or the dlib library.

[0024] In the normalization and frontal orientation process, the coordinate values ​​of each landmark point are transformed to match a predetermined standard face size. Even if the subject's face in the image captured by the imaging device 10 is tilted or small, the normalization and frontal orientation process ensures that the face always has a consistent size and is facing forward. This reduces the degradation of estimation accuracy caused by variations in the imaging device 10's field of view and the subject's face.

[0025] Returning to Figure 3, the drowsiness motion detection unit 32 will be described. The drowsiness motion detection unit 32 detects the driver's state from the actions expressed on the driver's face. For example, the drowsiness motion detection unit 32 determines the drowsiness level based on a predetermined rule (algorithm) for the image captured by the imaging device 10. The drowsiness motion detection unit 32 in this embodiment is composed of a frequent yawning detector 41, a yawning detector 42, a frequent slow blinking detector 43, and a slow blinking detector 44. In this embodiment, the description assumes that an imaging device 10 is provided to photograph the driver's face in order to extract the characteristics of the actions expressed on the driver's face, but the means for extracting the characteristics of the actions expressed on the driver's face are not limited to the imaging device 10. For example, instead of the imaging device 10, multiple sensors that detect the movement of the cheek muscles may be attached to the driver's face. In this case, the multiple sensors are connected to the drowsiness motion detection unit 32 via a communication line. The sensors are, for example, piezoelectric sensors.

[0026] The yawning frequency detector 41 and the yawning detector 42 will be explained with reference to Figure 5. Figure 5 is a diagram illustrating the process of detecting a subject's yawning.

[0027] Both the yawning frequency detector 41 and the yawning detector 42 detect yawns. The difference between the yawning frequency detector 41 and the yawning detector 42 is the number of yawns detected within a predetermined time. The yawning frequency detector 41 determines that the detection condition has been met when it detects a predetermined number of yawns or more within a predetermined time. The yawning frequency detector 41 determines that the detection condition has been met when it detects a yawn, regardless of the frequency of yawn detection.

[0028] The yawn detection process common to the yawning frequency detector 41 and the yawning detector 42 will now be described. In this embodiment, the yawn detection process calculates the degree of mouth opening based on the position of the mouth landmark point in the image, and detects yawns based on a change score that indicates the degree of change in the degree of mouth opening.

[0029] The change score can be calculated using a subspace method, for example, as shown in Figure 5. In the subspace method, the change score is calculated by scanning scan window 1 (present) and scan window 2 (past) in the time direction with respect to data showing the change in mouth opening over time. The positional relationship between the time-direction width W of scan window 1 and scan window 2, and the overlap width L of scan window 1 and scan window 2, is set, for example, to (W,L)=(150,60). By adopting this method, high robustness to noise and low processing load can be achieved without supervised learning, and high values ​​can be output in response to changes even with waveforms where the MAR (Mouth Aspect Ratio), which represents the degree of mouth opening, is small.

[0030] Furthermore, the width W and overlap width L of scanning window 1 and scanning window 2, respectively, are tuned as hyperparameters. The average duration of yawning is said to be 5.9 seconds, while in the experimental video it was 3-5 seconds. Therefore, the width W is set to 5 seconds, and the optimal value for the overlap width L is set based on the experimental video.

[0031] In the yawn detection process, if the calculated change score is higher than a pre-set threshold (for example, 0.45), it is determined that yawning has been detected.

[0032] The yawning frequency detector 41 transmits information to the sleepiness determination unit 34 indicating that the sleepiness level is 4 if the number of yawns detected within a predetermined time exceeds a predetermined number. The yawning detector 42 transmits information to the sleepiness determination unit 34 indicating that the sleepiness level is 3 if the number of yawns detected is less than a predetermined number and the number of yawns detected is one or more.

[0033] The slow blinking detection detector 43 and the slow blinking detection detector 44 will be explained with reference to Figure 6. Figure 6 is a diagram illustrating the process of detecting a subject's slow blinking.

[0034] Both the slow blinking detection detector 43 and the slow blinking detection detector 44 detect blinks. The difference between the slow blinking detection detector 43 and the slow blinking detection detector 44 is the number of blinks detected within a predetermined time. The slow blinking detection detector 43 determines that the detection condition has been met when it detects more than a predetermined number of blinks within a predetermined time. The slow blinking detection detector 44 determines that the detection condition has been met when it detects a blink, regardless of the blinking frequency.

[0035] The blink detection process common to the slow blinking detection detector 43 and the slow blinking detection detector 44 will now be described. In this embodiment, the blink detection process calculates the degree of eyelid opening based on the eye's landmark points (feature points), and determines that a slow blink has occurred if the eyelid opening is small for a certain period of time or longer. A "slow blink" is a blink in which the degree of eyelid opening is smaller than a predetermined degree for a predetermined period of time or longer.

[0036] As shown in Figure 6, the reference time is, for example, 11 frames (300 milliseconds) when the video recording frame rate is 30fps. In the detection process, if the time between the frame immediately before falling below the lower threshold and the frame immediately after exceeding the lower threshold is 11 frames or more, it is determined that a slow blink has occurred.

[0037] The slow blinking detection detector 43 transmits information to the drowsiness determination unit 34 indicating that the drowsiness level is 3 if the number of slow blinks detected within a predetermined time period is equal to or greater than a predetermined number. The slow blinking detection detector 44 transmits information to the drowsiness determination unit 34 indicating that the drowsiness level is 2 if the number of slow blinks detected is less than a predetermined number, and the number of slow blinks detected is 1 or more.

[0038] Next, the drowsiness estimation unit 33 will be described. The drowsiness estimation unit 33 estimates the degree of the driver's state from the actions expressed on the driver's face. The drowsiness estimation unit 33 is composed of a parameter acquisition unit 51, a drowsiness estimation model 52, and an estimated value calculation unit 53. In this embodiment, the drowsiness estimation unit 33 will be described in a case where the drowsiness estimation unit 33 estimates the degree of the driver's state from the actions expressed on the driver's face using the drowsiness estimation model 52 as a learning model, but estimation may also be performed without using a learning model.

[0039] The parameter acquisition unit 51 acquires input data from the images acquired by the imaging device 10 for input into the sleepiness estimation model 52.

[0040] Figure 7 is a table showing an example of input data for the sleepiness estimation model 52. Figure 7 shows five features as input data: the number of long blinks, the number of short blinks, the time the eyelids are partially closed, the frequency of long blinks, and the number of yawns. The number of features is not limited to the five shown in Figure 7; there may be six or more features.

[0041] The following explanation of each feature refers to Figures 8 to 12. In the following explanation, the output interval, which is also the granularity of the correct labels, is assumed to be 20 to 40 seconds, and the calibration period length when the beginning of the data is used for calibration is assumed to be 160 to 200 seconds. Eye opening is the harmonic mean of the left and right values ​​in each frame. Frames in which the eye opening is below the closed threshold indicate a closed eye state, and frames in which the eye opening is above the closed threshold but below the half-open threshold indicate a half-closed eye state. Blinking is a state in which the "closed" state continues for a certain number of frames or more.

[0042] Figure 8 is a graph illustrating the method for calculating the first feature as input data. The graph in Figure 8 shows the change in eye opening over time.

[0043] The long blink count is the first feature, representing the number of long blinks within the extraction period. The extraction period is the window size, which is the scanning period involved in the calculation, and is, for example, 20 to 40 seconds. A long blink here refers to a blink that exceeds the threshold value (blink_thrd) for the duration considered long. The threshold value is, for example, a judgment time of 0.2 to 0.4 seconds. A short blink is any blink other than a long blink. In other words, a short blink is a blink that is less than or equal to the judgment time.

[0044] The first feature is a parameter that indicates the number of blinks during the extraction period that consecutively fell below the "closing threshold" for a judgment time (e.g., 0.2 to 0.4 seconds) or longer. In this example, the first feature is 2.

[0045] Figure 9 is a graph illustrating the calculation method for the second feature as input data. The number of short blinks is the second feature, which indicates the number of short blinks during the extraction period. The second feature is a parameter that indicates the number of blinks during the extraction period (e.g., 20-40 seconds) where the time below the "closing threshold" is within the judgment time (e.g., 0.2-0.4 seconds). It is detected when the time the eyes were closed is shorter than the time the eyes were closed during the long blinks in the first feature. In this example, the second feature is 1.

[0046] Figure 10 is a graph illustrating the calculation method for the third feature as input data. The eyelid half-closed time is the third feature that represents the time tm of the eyelid half-closed state during the extraction period, which is the target time. The eyelid half-closed state is a state in which the eyelids are half closed. The third feature is a numerical value that represents the time proportional to the total number of frames in which the eyelids are below the half-open threshold and above the closed threshold within a predetermined time (for example, 20 to 40 seconds). In this example, the third feature tm is expressed as tm = (time per frame) × (number of frames). For example, if the time per frame is 0.033 seconds and the number of frames is 2, the time tm of the eyelid half-closed state is 0.066 seconds.

[0047] Figure 11 is a graph illustrating the calculation method for the fourth feature as input data. The frequency of long blinks is the fourth feature that indicates whether slow blinks occurred frequently during the frequency determination period, which includes the target time and past target time. The fourth feature is the sum of the number of long blinks during the extraction period as the target time (e.g., 20-40 seconds) and the number of long blinks during the pre-scan period as the prior time, which is a target time earlier than the target time (e.g., 20-40 seconds). The pre-scan period is a fixed period of time before the extraction time. If no long blinks occur during the extraction period or pre-scan period, the frequency of slow blinks is determined to be "none" or "did not occur." Figure 11 shows the case where the total number of long blinks occurs over two consecutive target time periods, but the two target time periods do not have to be consecutive.

[0048] Figure 12 is a graph illustrating the calculation method for the fifth feature as input data. The number of yawns is the fifth feature, which indicates the number of times a cluster of yawns appeared during the extraction period, which is the target time. The fifth feature is a numerical value that indicates the number of times the degree of mouth opening (MAR) was determined to be open within a predetermined time (e.g., 20 to 40 seconds). Mouth opening is determined to have occurred if the MAR continuously exceeds a determination threshold (e.g., 0.16 to 0.2) for a predetermined determination time (e.g., 0.5 to 0.7 seconds) or longer. In this example, the fifth feature is 1.

[0049] The drowsiness estimation model 52 is a learning model constructed through supervised learning. The drowsiness estimation model 52 is a learning model that uses information that quantifies blinking and mouth opening, based on facial feature points obtained from image data including the subject's face. For example, the drowsiness estimation model 52 is constructed by preparing a large dataset in which parameters, including the first to fifth features extracted from images including the subject's face, are used as input data, and drowsiness scores corresponding to the parameters are used as output data, and then performing machine learning on the dataset. The drowsiness score is, for example, a numerical value from 0 to 4 indicating the degree of drowsiness. The subject may be a driver.

[0050] The estimation unit 53 receives the first to fifth features obtained by the parameter acquisition unit 51 from the image actually captured by the imaging device 10 as input data. The estimation unit 53 then inputs the first to fifth features into the sleepiness estimation model 52 and outputs an estimated value indicating the sleepiness score as output data. The estimated value output from the estimation unit 53 is transmitted to the sleepiness determination unit 34.

[0051] The drowsiness determination unit 34 determines whether or not to intervene with the driver, such as issuing a warning, based on information indicating the driver's state, which is input from either or both of the drowsiness behavior detection unit 32 and the drowsiness estimation unit 33.

[0052] The drowsiness determination unit 34 determines whether or not to intervene with the driver based on the drowsiness level input from the drowsiness behavior detection unit 32. In this embodiment, the input of a drowsiness level from the drowsiness behavior detection unit 32 is the condition for intervening with the driver. The criteria for determining whether or not the drowsiness determination unit 34 intervenes with the driver may be set in advance, or it may be set by the driver. For example, the unit may be configured to intervene with the driver when the drowsiness level is 3 or higher, or when the drowsiness level is 4 or higher. Furthermore, the drowsiness determination unit 34 may be configured to intervene with the driver when multiple different drowsiness levels are input, such as when drowsiness levels 2 and 4 are input.

[0053] Furthermore, the drowsiness detection unit 34 determines whether or not to intervene with the driver based on the estimated value input from the estimated value calculation unit 53. The drowsiness detection unit 34 intervenes if the estimated value is above a preset threshold. The estimated value is a score such as estimated level 0 to 4.

[0054] The drowsiness detection unit 34 may determine whether or not to intervene with the driver based on the temporal changes in the drowsiness level or estimated value. For example, the drowsiness detection unit 34 may intervene with the driver if the drowsiness level has risen or the estimated value has increased after a predetermined period of time has elapsed.

[0055] Next, we will return to Figure 3 and explain the configuration of the notification device 2. The notification device 2 is a device that has a notification function, which includes a notification processing unit 61 controlled by the drowsiness detection unit 34. As described above, the notification function is a function that alerts the driver that drowsiness is occurring through sound or images.

[0056] The notification processing unit 61 executes a process to activate a display and speaker in order to alert the subject with images and sounds, based on the determination result of the drowsiness determination unit 34.

[0057] The notification processing unit 61 may change the level of alert according to the degree of drowsiness indicated by the drowsiness determination result of the drowsiness determination unit 34. For example, the notification processing unit 61 may perform notification processing such that the image display manner, volume, and content of communication to the target person become stronger in the order of drowsiness level 2, drowsiness level 3, and drowsiness level 4. Similarly, the notification processing unit 61 may perform notification processing such that the image display manner, volume, and content of communication to the target person become stronger the larger the estimated value calculated by the drowsiness estimation model 52.

[0058] Furthermore, when the notification processing unit 61 determines whether or not to intervene based on the temporal change in the drowsiness level or estimated value, it may change the intensity of the warning if the drowsiness level or estimated value increases over time. For example, if the drowsiness level or estimated value is higher than it was a predetermined time ago, the notification processing unit 61 may perform notification processing to warn the driver at a higher level than the warning level set for the drowsiness level or estimated value. In this processing as well, changing to a higher level of warning can be achieved by increasing the volume in the case of voice or changing the expression to a stronger announcement. In the case of images, changing to a higher level of warning can be achieved by changing the display manner, such as flashing, changing the color, or changing the text display.

[0059] <Processing flow> Next, with reference to Figure 13, the processing flow of the subject status determination system 100 will be described. Figure 13 is a flowchart showing an example of the processing flow of the subject status determination system 100 in this embodiment.

[0060] In step S1, the imaging device 10 acquires subject images (video) of the driver's face continuously captured while the vehicle 101 is in motion, and transmits them to the subject status determination device 1. In this embodiment, the imaging device 10 transmits video data captured at 30-second intervals while the vehicle 101 is in motion to the subject status determination device 1.

[0061] In step S2, the preprocessing unit 31 of the subject status determination device 1 performs normalization and frontalization processing on the image acquired from the imaging device 10.

[0062] In step S3, the yawning frequency detector 41 of the drowsiness motion detection unit 32 counts the number of yawns detected from the image after normalization and frontal processing, and determines whether the number of yawns detected within a predetermined time is equal to or greater than a predetermined number. If the number of yawns detected within the predetermined time is equal to or greater than a predetermined number, the yawning frequency detector 41 proceeds to the process of step S9 described later (step S3; Yes). On the other hand, if the number of yawns detected within the predetermined time is less than a predetermined number, the yawning frequency detector 41 proceeds to the process of step S4 (step S3; No).

[0063] In step S4, the yawn detector 42 determines whether the number of detected yawns within a predetermined time is less than a predetermined number and whether there is at least one yawn. If the determination shows that there is at least one yawn, the yawn detector 42 proceeds to the process in step S9 described later (step S4; Yes). On the other hand, if the determination shows that no yawns are detected, the yawn detector 42 proceeds to the process in step S5 (step S4; No).

[0064] In step S5, the slow blink detection detector 43 counts the number of slow blinks detected from the image after normalization and frontalization processing, and determines whether the number of slow blinks detected within a predetermined time is equal to or greater than a predetermined number. If the number of detections within a predetermined time is equal to or greater than a predetermined number, the slow blink detection detector 43 proceeds to the process in step S9 described later (step S5; Yes). On the other hand, if the number of detections within a predetermined time is less than a predetermined number, the slow blink detection detector 43 proceeds to the process in step S6 (step S5; No).

[0065] In step S6, the slow blink detector 44 determines whether the number of slow blinks detected within a predetermined time is less than a predetermined number and whether there is at least one slow blink. If there is at least one slow blink detected, the slow blink detector 44 proceeds to the process in step S9 described later (step S6; Yes). On the other hand, if no slow blinks are detected, the slow blink detector 44 proceeds to the process in step S7 (step S6; No).

[0066] In step S7, the parameter acquisition unit 51 of the sleepiness estimation unit 33 acquires parameters (first to fifth features) from the image after normalization and frontalization processing to be input to the sleepiness estimation model 52.

[0067] In step S8, the estimated value calculation unit 53 inputs the parameters acquired by the parameter acquisition unit 51 into the sleepiness estimation model 52, outputs an estimated value, and proceeds to the process in step S9.

[0068] In step S9, the drowsiness detection unit 32 and the drowsiness estimation unit 33 transmit information indicating the degree of drowsiness as the driver's state to the drowsiness determination unit 34. If the judgment conditions of the frequent yawning detector 41 are met, the information indicating the degree of drowsiness will be drowsiness level 2, and if the judgment conditions of the yawning detector 42 are met, the information indicating the degree of drowsiness will be drowsiness level 3. In addition, if the judgment conditions of the frequent slow blinking detector 43 are met, the information indicating the degree of drowsiness will be drowsiness level 3, and if the judgment conditions of the slow blinking detector 44 are met, the information indicating the degree of drowsiness will be drowsiness level 4. Furthermore, if none of the judgment conditions of the frequent yawning detector 41, the yawning detector 42, the frequent slow blinking detector 43, and the slow blinking detector 44 are met, the estimated value calculated by the estimated value calculation unit 53 is transmitted to the drowsiness determination unit 34 as information indicating the degree of drowsiness.

[0069] In step S10, the drowsiness determination unit 34 determines whether or not to intervene with the driver based on information indicating the degree of drowsiness input from the drowsiness behavior detection unit 32 or the drowsiness estimation unit 33. In this embodiment, if the drowsiness determination unit 34 receives a drowsiness level input from the drowsiness behavior detection unit 32, or if the estimated value input from the drowsiness estimation model 52 is greater than or equal to a threshold, it determines that it should intervene with the driver and proceeds to the process in step S11 (step S10; Yes). If the drowsiness determination unit 34 does not receive a drowsiness level input from the drowsiness behavior detection unit 32, and the estimated value input from the drowsiness estimation model 52 is less than a threshold (for example, score 0), it determines that it should not intervene with the driver and returns to the process in step S1 (step S10; No).

[0070] In step S11, the notification processing unit 61 performs an audio or visual notification to alert the driver that drowsiness is occurring. The level of this notification may be increased as the drowsiness level or estimated value increases, or it may be performed at a uniform level regardless of the drowsiness level or the height of the estimated value above the threshold. Furthermore, the notification processing unit 61 may refer to past drowsiness levels or estimated values ​​and control the level of the notification to be higher than normal if the degree of drowsiness is higher than in the past.

[0071] In this embodiment, the subject state determination device 1 was described as determining that the subject is experiencing drowsiness as a specific state, but the specific state is not limited to the state of experiencing drowsiness. For example, the subject state determination device 1 may be used for disease diagnosis or healthcare. It is conceivable that the subject state determination device 1 could determine a medical condition such as facial neuralgia as a specific state based on facial movements such as twitching of the patient's cheek. It is also conceivable that the subject state determination device 1 could estimate the mental stress on the patient based on facial movements and determine a medical condition such as depression as a specific state.

[0072] As described above, the subject state determination system 100 of this embodiment includes a drowsiness motion detection unit 32 (motion detection unit) that detects the subject's state from the motions expressed on the subject's face, a drowsiness estimation unit 33 (estimation unit) that estimates the degree of the subject's state from the motions expressed on the subject's face, and a drowsiness determination unit 34 (determination unit) that determines whether the subject is in a specific state based on either or both of the detection result of the drowsiness motion detection unit 32 and the estimation result of the drowsiness estimation unit 33. When the motion expressed on the subject's face is an eye motion, it includes the subject's gaze and eye movements including the upper and lower eyelids.

[0073] Furthermore, the subject state determination method of this embodiment includes: an action detection step of detecting the subject's state from actions expressed on the subject's face; an estimation step of estimating the degree of the subject's state from actions expressed on the subject's face; and a determination step of determining whether the subject is in a specific state based on one or both of the detection result of the action detection step and the estimation result of the estimation step.

[0074] Furthermore, the program of this embodiment causes a computer to execute a motion detection step that detects the state of the subject from the movements expressed on the subject's face, an estimation step that estimates the degree of the subject's state from the movements expressed on the subject's face, and a determination step that determines whether or not the subject is in a specific state based on one or both of the detection result of the motion detection step and the estimation result of the estimation step.

[0075] According to this embodiment, a specific state (a state of drowsiness) occurring in a subject can be reliably determined by two means: detection using a pre-set rule-based algorithm and estimation using machine learning or the like.

[0076] Furthermore, the specific state in this embodiment is a state in which the subject is experiencing drowsiness. This makes it possible to reliably detect drowsiness in the subject in areas where drowsiness could lead to serious accidents, such as while driving, thereby preventing accidents from occurring.

[0077] Furthermore, the drowsiness detection unit 32 of this embodiment includes a frequent yawning detector 41 and a yawning detector 42 that detect yawns based on mouth movements, and a frequent slow blinking detector 43 and a slow blinking detector 44 that detect slow blinks based on eye movements. It detects that the subject is experiencing drowsiness based on one or both of the detection results of the frequent yawning detector 41 and the yawning detector 42, and one or both of the detection results of the frequent slow blinking detector 43 and the slow blinking detector 44. This makes it possible to reliably detect yawns and blinks that indicate drowsiness on a rule-based basis.

[0078] Furthermore, the drowsiness detection unit 32 of this embodiment detects the drowsiness level, which indicates the degree of drowsiness of the subject, based on either the number of detected yawns or the number of detected slow blinks, or both. This makes it possible to determine whether or not to intervene (notify the subject) according to the drowsiness level, or to determine the degree of intervention according to the drowsiness level, and to take measures according to the degree of drowsiness of the subject.

[0079] Furthermore, the subject state determination system 100 of this embodiment further includes a parameter acquisition unit 51 that acquires multiple feature quantities as parameters for the actions expressed on the subject's face, and the subject image data and subject image data are data containing multiple feature quantities. As a result, the degree of a specific state of the subject (an estimated value indicating drowsiness) can be calculated with high accuracy through supervised learning.

[0080] Furthermore, the subject state determination system 100 of this embodiment further includes a notification device 2 that notifies the subject of drowsiness by sound or image when the subject is determined to be in a specific state based on either or both of the detection results of the drowsiness action detection unit 32 and the estimation results of the drowsiness estimation unit 33, and the notification device 2 changes the volume of the notification or the display mode of the image to the subject according to the degree of drowsiness. As a result, the subject can be warned with a volume of sound or an image display mode that corresponds to the degree of danger indicated by the degree of drowsiness, thereby more reliably preventing accidents caused by drowsiness.

[0081] Although one embodiment of the present invention has been described above, the present invention is not limited to the embodiments described above, and any modifications, improvements, etc. that can achieve the objectives of the present invention are included in the present invention.

[0082] The subject state determination device 1 in the above embodiment is configured to determine that the subject is experiencing drowsiness as a specific state, but the specific state is not limited to the state of drowsiness. For example, the subject state determination device 1 may determine a medical condition as a specific state based on the condition of the eyes and mouth on the subject's face.

[0083] Furthermore, the series of processes described above can be executed by hardware or by software. In other words, the functional configuration described above is merely illustrative and not particularly limiting. That is, it is sufficient for the subject status determination system 100 to be equipped with a function that can execute the series of processes described above as a whole, and the type of functional block used to realize this function is not particularly limited to the example described above. Also, the location of the functional block is not particularly limited and can be arbitrary. For example, the functional block of the subject status determination device 1 may be transferred to another device, etc. Conversely, the functional block of another device may be transferred to another configuration of the subject status determination device 1, etc. Also, a single functional block may be composed of hardware alone, software alone, or a combination of both.

[0084] When a series of processes are executed by software, the programs that make up that software are installed on a computer or other device from a network or storage medium. The computer may be a computer built into dedicated hardware. Alternatively, the computer may be a computer capable of performing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0085] Such recording media containing programs may consist not only of removable media (not shown) distributed separately from the main device to provide the programs, but also of recording media provided pre-installed in the main device. Since programs can be distributed via a network, the recording media may be installed on or accessible from a computer connected to or capable of connecting to a network.

[0086] In this specification, the step of describing a program to be recorded on a recording medium includes not only processes that are performed chronologically in that order, but also processes that are not necessarily performed chronologically, but are executed in parallel or individually. Furthermore, in this specification, the term "system" refers to an overall system composed of multiple devices, means, etc. [Explanation of symbols]

[0087] 1. Subject Status Determination Device 2. Notification device 10 Imaging device 32 Sleepiness detection unit 33 Sleepiness estimation unit 41. Frequent Yawning Detector 42 Yawning Detector 43 Slow blinking frequent detector 44 Slow blinking detector 100 Target Person Status Determination System

Claims

1. A motion detection unit that detects the state of the subject from the actions expressed on the subject's face, An estimation unit that estimates the degree of the subject's condition from the actions expressed on the subject's face, A determination unit that determines whether the subject is in a specific state based on either or both of the detection result of the motion detection unit and the estimation result of the estimation unit, A system for determining the status of a person having the following characteristics.

2. The aforementioned motion detection unit, The aforementioned facial features include a yawn detector that detects yawning based on mouth movements, A slow blink detector, which detects slow blinks among the aforementioned faces based on eye movements, where the amount of time the eyelids are open is less than a predetermined amount is longer than a predetermined amount, It has, Based on the detection result of the yawning detector and the detection result of the slow blinking detector, or both, it is detected that the subject is in the specific state. The subject status determination system according to claim 1.

3. The aforementioned motion detection unit, A state level indicating the degree of the subject's specific state is detected based on either or both of the number of detected yawns and the number of detected slow blinks. The subject status determination system according to claim 2.

4. The device further comprises a notification device that, when the subject is determined to be in the specific state based on either or both of the detection result of the motion detection unit and the estimation result of the estimation unit, notifies the subject that the specific state has occurred by voice or image, The aforementioned notification device is The volume or display mode of the image notified to the subject is changed according to the degree of the aforementioned specific condition. The subject status determination system according to claim 2 or 3.

5. The aforementioned specific state is a state in which the subject is experiencing drowsiness. The subject status determination system according to claim 4.

6. A method for determining the status of a subject, which is performed by a computer that determines the status of the subject, A motion detection step for detecting the state of the subject from the actions expressed on the subject's face, An estimation step of estimating the degree of the subject's condition from the actions expressed on the subject's face, A determination step that determines whether the subject is in a specific state based on either or both of the detection result of the motion detection step and the estimation result of the estimation step, A method for determining the status of a subject having the following characteristics.

7. A motion detection step that detects the state of the subject from the actions expressed on the subject's face, An estimation step of estimating the degree of the subject's condition from the actions expressed on the subject's face, A determination step that determines whether the subject is in a specific state based on either or both of the detection result of the motion detection step and the estimation result of the estimation step, A program that causes a computer to execute something.

Images