Face fatigue detection method of multi-feature vector machine

Abstract

According to the face fatigue detection method based on the multi-feature vector machine, feature extraction is carried out on an input face image sequence by utilizing Dlib face feature positioning, and mouth feature classification and eye feature classification are carried out by utilizing face information; and further training the classified and labeled eye and mouth data sets through a support vector machine. And finally, applying the trained classifier to fatigue recognition, and inputting a result of the classifier into a fatigue detection method to judge a fatigue state. According to the method, generalization of the training model is enhanced in a data driving mode, the fatigue detection capability of the model in different scenes is improved, and the method has the advantages of low cost, low invasiveness and high accuracy and can be applied to fatigue detection tasks of related personnel in different industrial scenes.

Description

Technical Field

[0001] The invention belongs to the field of computer vision face image algorithm recognition, and specifically relates to a face fatigue detection method of a multi-feature vector machine. Background Art

[0002] As an important issue in the field of computer vision, face fatigue detection algorithms are applied in various industries, such as traffic fatigue driving (such as cars, public transportation and airplanes), factory worker fatigue work (such as auto repair workers, civil construction workers, etc.), and medical care (nurses, care workers and emergency room doctors). In the above application scenarios, the operator's work efficiency will decrease as the degree of fatigue increases, and even danger may occur. Fatigue detection technology can prevent and warn them. The original intention of the face fatigue detection algorithm is to effectively monitor the fatigue state of people, thereby improving safety and efficiency in various industries.

[0003] Existing facial fatigue detection methods are mainly divided into two categories: one is to conduct research from the perspective of biological physiological signal feature extraction, and judge the target's fatigue level by analyzing physiological parameters such as ECG signals and EEG signals. This type of method has extremely high fatigue recognition accuracy, but in actual social production, the high cost of equipment and the complicated process of target detection limit the application of fatigue detection of this type of method, and the detection instrument may interfere with normal operation and has high invasiveness; another type is to start from the perspective of facial feature extraction based on computer vision, and perform fatigue detection by visually monitoring the pupil dilation state and subtle changes in medical facial PPG signals. However, the application scope of this method is relatively limited, and it is difficult to meet the detection needs of different races, postures, light and environments. For example, in an environment with large light changes, the detection of pupil state may be disturbed. Differences in facial features between different races may affect the detection results. Among the computer vision-based methods, there is also a method that detects the target's eye state and judges the sleepiness state by the time the eyes are open and closed. This method has a high accuracy rate in practical applications, and has the advantages of low cost and low invasiveness, and can effectively analyze the eye state. Therefore, it has been widely adopted in practical applications and has become one of the mainstream methods for fatigue detection. However, this method has certain limitations, such as a single judgment factor, which mainly relies on eye activity and fails to fully reflect the fatigue state. In addition, the formula for judging the eye state has poor applicability and lacks flexibility, making it difficult to adapt to complex and changing actual scenarios.

[0004] Through the above research status analysis, we believe that the current fatigue detection methods have the following shortcomings: the existing fatigue detection methods, in the process of extracting biological physiological signals, the detection equipment is expensive, the detection process is complicated and highly invasive, which will affect the normal operation of the detection target; in addition, the mainstream detection methods mainly focus on the study of the state of human face eyes, and the detection model only has PERCLOS (Percentage of Eyelid Closure over the Pupilover Time, PERCLOS), and the joint detection of mouth features is not associated. In addition, the classification method for judging the state of mouth and eye features lacks universality and does not take into account the individual differences of the monitoring targets, which may cause misjudgment in complex scenarios. Summary of the invention

[0005] In order to overcome the deficiencies in the existing solutions, the purpose of the present invention is to provide a low-cost, low-invasive and high-accuracy multi-feature vector machine face fatigue detection method.

[0006] The technical solution adopted by the present invention is a method for detecting facial fatigue using a multi-feature vector machine, comprising the following steps:

[0007] Step 1: Start the camera, read the camera video frame, and use the preset Dlib method to locate the face points. After obtaining the fixed 68 points, calculate the Euclidean distance formula of the fixed points according to formula (1):

[0008]

[0009] Where P and Q are any two points in the face. x ,P y ) and (Q x ,Q y ) are the coordinates of these two points respectively;

[0010] Step 2: Calculate the eye aspect ratio mEAR according to the point information obtained in step 1 according to formula (2):

[0011]

[0012] Among them (P 43 ,P 46 ) are the two endpoints of the eye horizontal direction, (P 44 ,P 48 ) and (P 45 ,P 47 ) are two pairs of endpoints in the vertical direction of the eye;

[0013] Step 3: Calculate the mouth length-to-width ratio mMAR according to the point information obtained in step 1 according to formula (3):

[0014]

[0015] Among them (P 49 ,P 55 ) are the two endpoints of the mouth in the horizontal direction, (P 51 ,P 59 ) and (P 53 ,P 57 ) are the two pairs of endpoints of the mouth in the vertical direction;

[0016] Step 4: Read information from each frame of the video. According to formula (4), if mEAR is greater than the threshold T e Then Eflag i +1 otherwise -1, if mMAR is greater than the set threshold T m Mflag i +1 otherwise -1,

[0017]

[0018] Where i is the frame processing sequence number;

[0019] Step 5: Mark the video frame image set according to step 4. If Eflag i = +1, then the unique feature set of the open-eye image sequence {Eo i}, if Eflagi=-1, then the closed-eye image feature set {Ec i}, similarly, Mflagi=+1 marks the closed mouth image feature set {Mc i}, if Mflag i = -1, then mark the mouth image feature set {Mo i}, according to formula (5), we get the eye image set E i ′,1≤i≤Max and mouth image set M i ′,1≤i≤Max,

[0020]

[0021] Among them, i is the image number, the maximum value is Max, and ∪ is the parallel symbol;

[0022] Step 6, for step 5, we get {E i The dataset is normalized according to formula (6) to obtain the training dataset for eye-opening and eye-closing classification.

[0023]

[0024] Among them, E i ∈R d is the input sample, Ei ′ is the original data set, μ is the mean of the data, σ is the standard deviation of the data, E i is the normalized data set, Eflag i ∈{+1,-1} is E i The corresponding label, if Eflag i = +1, then assume Eflag i For positive samples, if Eflag i = -1, then assume Eflag i is a negative sample;

[0025] Step 7: train the data set obtained in step 6 and obtain an optimal hyperplane D for eyes open and eyes closed according to formula (7).

[0026]

[0027] Where D is the optimal hyperplane, S is the minimum vector distance, w is the hyperplane normal vector, and w T represents the transpose of the normal vector, and b is the intercept;

[0028] Step 8: According to the optimal separation hyperplane and decision function calculated by formula (8), a unique eye feature classifier is obtained.

[0029]

[0030] Among them, sign is the sign function, w * is the optimal hyperplane, b * is the optimal intercept, x is the input vector;

[0031] Step 9, follow steps 6, 7 and 8 to obtain a unique mouth feature classifier, then initialize the eye counter EC and mouth counter MO. If the eyes are closed, EC increases by 1, and if the mouth is open, MO increases by 1.

[0032] Step 10: EC and MO obtained in step 9 are used to determine the fatigue detection model according to formula (9).

[0033]

[0034] Where PE is eye fatigue, PM is mouth fatigue, and t is the fixed interval time;

[0035] Step 11, calculate the fatigue index according to formula (10) to get FD,

[0036] FD=2-PE-PM (10)

[0037] When FD is less than 1.6, an alarm is issued to remind that the vehicle is in a fatigue state. When FD is less than 1.4, an alarm is issued to remind that the vehicle is in a severe fatigue state.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] (1) The present invention proposes a multi-feature vector machine for detecting the association of eye and mouth features, which can effectively associate and obtain facial fatigue information of a human face, thereby improving the detection accuracy of facial fatigue under the reference system of the association of eye and mouth features. In addition, the face detection method based on the multi-feature vector machine is simple and portable, and has the characteristics of low cost and low invasiveness.

[0040] (2) The present invention adopts a data-driven approach to pre-train the facial information through machine learning, providing a more flexible method for processing user self-feedback. In addition, the method for judging the state of the eyes and mouth is transformed from the traditional hard threshold to a soft threshold that adapts to different application scenarios, effectively enhancing the accuracy of the facial fatigue detection method in different environments, different individuals, and different postures, and improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of a method for detecting facial fatigue using a multi-feature vector machine;

[0042] Figure 2 It is a bitmap of 68 information points of Dlib's face in a face fatigue detection method using a multi-feature vector machine;

[0043] Figure 3 It is an eye detection point diagram of a face fatigue detection method using a multi-feature vector machine;

[0044] Figure 4 A mouth detection point diagram of a face fatigue detection method using a multi-feature vector machine;

[0045] Figure 5 This is a diagram of eye feature opening and closing changes in a multi-feature vector machine facial fatigue detection method;

[0046] Figure 6 This is a mouth feature opening and closing change diagram of a face fatigue detection method based on a multi-feature vector machine. DETAILED DESCRIPTION

[0047] Example 1

[0048] like Figure 1 As shown, a method for detecting facial fatigue using a multi-feature vector machine comprises the following steps:

[0049] Step 1: Start the camera, read the camera video frame, and use the preset Dlib method to locate the face points. After obtaining the fixed 68 points, Figure 2 As shown, according to formula (1), the Euclidean distance formula of the fixed point is calculated:

[0050]

[0051] Where P and Q are any two points in the face. x ,P y ) and (Q x ,Q y ) are the coordinates of these two points respectively;

[0052] Step 2: Calculate the eye aspect ratio mEAR according to the point information obtained in step 1 according to formula (2): Figure 3 As shown,

[0053]

[0054] Among them (P 43 ,P 46 ) are the two endpoints of the eye horizontal direction, (P 44 ,P 48 ) and (P 45 ,P 47 ) are two pairs of endpoints in the vertical direction of the eye;

[0055] Step 3: Calculate the mouth length-to-width ratio mMAR according to the point information obtained in step 1 according to formula (3): Figure 4 As shown,

[0056]

[0057] Among them (P 49 ,P 55 ) are the two endpoints of the mouth in the horizontal direction, (P 51 ,P 59 ) and (P 53 ,P 57 ) are the two pairs of endpoints of the mouth in the vertical direction;

[0058] Step 4: Read information from each frame of the video. According to formula (4), if mEAR is greater than the threshold T e Then Eflag i +1 otherwise -1, if mMAR is greater than the set threshold T m Mflag i +1 otherwise -1,

[0059]

[0060] Where i is the frame processing sequence number;

[0061] Step 5: Mark the video frame image set according to step 4. If Eflag i = +1, then the unique feature set of the open-eye image sequence {Eo i}, if Eflagi=-1, then the closed-eye image feature set {Ec i}, similarly, Mflagi=+1 marks the closed mouth image feature set {Mc i}, if Mflag i = -1, then mark the mouth image feature set {Mo i}, according to formula (5), we get the eye image set E i ′,1≤i≤Max and mouth image set M i ′,1≤i≤Max,

[0062]

[0063] Among them, i is the image number, the maximum value is Max, and ∪ is the parallel symbol;

[0064] Step 6, for step 5, we get {E i The dataset is normalized according to formula (6) to obtain the training dataset for eye-opening and eye-closing classification.

[0065]

[0066] Among them, E i ∈R d is the input sample, E i ′ is the original data set, μ is the mean of the data, σ is the standard deviation of the data, E i is the normalized data set, Eflag i ∈{+1,-1} is E i The corresponding label, if Eflag i = +1, then assume Eflag i For positive samples, if Eflag i = -1, then assume Eflag i is a negative sample;

[0067] Step 7: train the data set obtained in step 6 and obtain an optimal hyperplane D for eyes open and eyes closed according to formula (7).

[0068]

[0069] Where D is the optimal hyperplane, S is the minimum vector distance, w is the hyperplane normal vector, and w T represents the transpose of the normal vector, and b is the intercept;

[0070] Step 8: According to the optimal separation hyperplane and decision function calculated by formula (8), a unique eye feature classifier is obtained.

[0071]

[0072] Among them, sign is the sign function, w * is the optimal hyperplane, b * is the optimal intercept, x is the input vector;

[0073] Step 9, follow steps 6, 7 and 8 to obtain a unique mouth feature classifier, then initialize the eye counter EC and mouth counter MO. If the eyes are closed, EC increases by 1, and if the mouth is open, MO increases by 1.

[0074] Step 10, EC and MO obtained in step 9 are then used to determine the fatigue detection model according to formula (9), as follows: Figure 5 and Figure 6 As shown,

[0075]

[0076] Where PE is eye fatigue, PM is mouth fatigue, and t is the fixed interval time;

[0077] Step 11, calculate the fatigue index according to formula (10) to get FD,

[0078] FD=2-PE-PM (10)

[0079] When FD is less than 1.6, an alarm is issued to remind that the vehicle is in a fatigue state. When FD is less than 1.4, an alarm is issued to remind that the vehicle is in a severe fatigue state.

[0080] In summary, the present invention is aimed at the joint detection of visual facial features, starting from the extraction of facial features, comprehensively detecting facial fatigue features, deepening facial information analysis, and monitoring and analyzing the status of the eyes and mouth in real time. The use of computer vision methods ensures low intrusion and low cost. In addition, compared with the traditional hard threshold classification, this patent designs a facial fatigue detection method of a multi-feature vector machine, which performs online training at the beginning of the detection, extracts the eye and mouth features of the target at a fixed number of frames, and uses the support vector machine method to classify the eye and mouth features, and then further uses the fatigue model for evaluation. This method has the characteristics of low cost, low intrusion, high accuracy and high robustness.

Claims

1. A method for detecting facial fatigue using a multi-feature vector machine, characterized in that: The following steps are involved: Step 1: Start the camera, read the camera video frame, and use the preset Dlib method to locate the face points. After obtaining the fixed 68 points, calculate the Euclidean distance formula of the fixed points according to formula (1): Where P and Q are any two points in the face. x ,P y ) and (Q x ,Q y ) are the coordinates of these two points respectively; Step 2: Calculate the eye aspect ratio mEAR according to the point information obtained in step 1 according to formula (2): Among them (P 43 ,P 46 ) are the two endpoints of the eye horizontal direction, (P 44 ,P 48 ) and (P 45 ,P 47 ) are two pairs of endpoints in the vertical direction of the eye; Step 3: Calculate the mouth length-to-width ratio mMAR according to the point information obtained in step 1 according to formula (3). Among them (P 49 ,P 55 ) are the two endpoints of the mouth in the horizontal direction, (P 51 ,P 59 ) and (P 53 ,P 57 ) are the two pairs of endpoints of the mouth in the vertical direction; Step 4: Read information from each frame of the video. According to formula (4), if mEAR is greater than the threshold T e Then Eflag i +1 otherwise -1, if mMAR is greater than the set threshold T m Mflag i +1 otherwise -1, Where i is the frame processing sequence number; Step 5: Mark the video frame image set according to step 4. If Eflag i = +1, then the unique feature set of the open-eye image sequence {Eo i }, if Eflag i = -1, then the closed-eye image feature set {Ec i }, similarly, Mflag i = +1 mark closed mouth image feature set {Mci}, if Mflag i = -1, then mark the mouth image feature set {Moi}, and obtain the eye image set E according to formula (5) i ′,1≤i≤Max and mouth image set M i ′,1≤i≤Max, Among them, i is the image number, the maximum value is Max, and ∪ is the parallel symbol; Step 6, get {E i The dataset is normalized according to formula (6) to obtain the training dataset for eye-opening and eye-closing classification. Among them, E i ∈R d is the input sample, E i ′ is the original data set, μ is the mean of the data, σ is the standard deviation of the data, E i is the normalized data set, Eflag i ∈{+1,-1} is E i The corresponding label, if Eflag i = +1, then assume Eflag i For positive samples, if Eflag i = -1, then assume Eflag i is a negative sample; Step 7: train the data set obtained in step 6 and obtain an optimal hyperplane D for eyes open and eyes closed according to formula (7). Where D is the optimal hyperplane, S is the minimum vector distance, w is the hyperplane normal vector, and w T represents the transpose of the normal vector, and b is the intercept; Step 8: According to the optimal separation hyperplane and decision function calculated by formula (8), a unique eye feature classifier is obtained. Among them, sign is the sign function, w * is the optimal hyperplane, b * is the optimal intercept, x is the input vector; Step 9, follow steps 6, 7 and 8 to obtain a unique mouth feature classifier, then initialize the eye counter EC and mouth counter MO. If the eyes are closed, EC increases by 1, and if the mouth is open, MO increases by 1. Step 10: EC and MO obtained in step 9 are used to determine the fatigue detection model according to formula (9). Where PE is eye fatigue, PM is mouth fatigue, and t is the fixed interval time; Step 11, calculate the fatigue index according to formula (10) to get FD, FD=2-PE-PM (10) When FD is less than 1.6, an alarm is issued to remind that the vehicle is in a fatigue state. When FD is less than 1.4, an alarm is issued to remind that the vehicle is in a severe fatigue state.

Images