Adaptive fatigue state detection method based on multi-modal feature fusion

Abstract

The invention discloses a self-adaptive fatigue state detection method based on multi-modal feature fusion, and belongs to the technical field of fatigue detection, and the method comprises the steps: synchronously collecting facial images through an infrared camera and an RGB camera, carrying out the image fusion through a Laplacian pyramid method, extracting facial features, setting a personalized threshold value, and obtaining fatigue features. Performing special scene judgment under limited face recognition, considering micro-expression features, performing fatigue parameter supplementation, judging a fatigue state through a comprehensive fatigue model with multi-fatigue feature fusion, and performing early warning information reminding; according to the adaptive fatigue state detection method based on multi-modal feature fusion, through multi-modal feature fusion, image details are effectively reserved, a low-light environment is adapted, facial feature deficiency is compensated, and the accuracy and stability of fatigue state detection are remarkably improved by combining personalized threshold values and special scene judgment.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue detection, and in particular to an adaptive fatigue state detection method based on multi-modal feature fusion. Background Art

[0002] In today's social production and transportation, production safety accidents caused by fatigue occur frequently, which have a serious impact on the lives, safety and health of operators. In addition, accidents caused by fatigue driving account for a relatively high proportion in traffic accidents, posing a serious threat to traffic safety. In this context, it is crucial to strengthen the detection of fatigue driving and the identification of employee fatigue.

[0003] Currently, fatigue detection technologies can be roughly divided into three categories: physiological signal fatigue recognition method, operation behavior feature fatigue recognition method, and human visible feature fatigue recognition method.

[0004] Existing fatigue detections using human visible features mostly rely on a single sensor (such as an RGB camera) or a single physiological feature (such as the eye closure frequency), and the detection accuracy drops significantly in complex scenarios (where the face cannot be recognized or is occluded). In addition, traditional methods do not fully consider the influence of individual feature differences on fatigue detection, resulting in a high false alarm rate. Therefore, there is an urgent need for a fatigue detection scheme that can adapt to multiple scenarios, fuse multi-dimensional features and has high robustness. Summary of the Invention

[0005] The purpose of the present invention is to provide an adaptive fatigue state detection method based on multi-modal feature fusion to solve the problems mentioned in the above background art.

[0006] To achieve the above purpose, the present invention provides an adaptive fatigue state detection method based on multi-modal feature fusion, including the following steps:

[0007] S1. Use an infrared camera and an RGB camera to synchronously collect facial images, and perform image fusion through the Laplacian pyramid method;

[0008] S2. Extract fatigue feature data of the eyes, mouth and micro-expression from the fused facial images, set personalized thresholds, and obtain fatigue features;

[0009] S3. When the face is occluded or the recognition is limited, use a posture classification model to identify whether it is a head-down sleeping scenario. If it is determined to be a head-down sleeping scenario, input the head-down sleeping features into the S5 fatigue state judgment module as the basis for fatigue detection;

[0010] S4. Under the mouth occlusion state, consider the micro-expression fatigue features for fatigue parameter compensation;

[0011] S5. Construct a comprehensive fatigue model that fuses multiple fatigue features to judge the fatigue state, which is divided into awake, fatigued, and severely fatigued states.

[0012] S6. When a fatigued or severely fatigued state is detected, a fatigue warning message is sent for reminder.

[0013] Preferably, the specific steps of S1 are as follows:

[0014] S11. Simultaneously collect facial images through an infrared camera and an RGB camera to obtain an RGB image I RGB and an infrared image I IR . Align the infrared image with the RGB image through geometric transformations: rotation and scaling, and adjust them to the same resolution.

[0015] S12. Construct Gaussian pyramids for the infrared image and the RGB image respectively to obtain their respective Gaussian pyramid sequences. The number of layers N of the Gaussian pyramid is determined according to the image size and requirements. The formula is as follows:

[0016]

[0017] Where, and represent the K-th layer Gaussian pyramid images of the RGB image and the infrared image respectively. Reduce is the Gaussian blur and downsampling operation.

[0018] S13. Construct Laplacian pyramids. Construct Laplacian pyramids for the Gaussian pyramids of the RGB image and the infrared image respectively. The formula is as follows:

[0019]

[0020] Where, and represent the K-th layer Laplacian pyramid images of the RGB image and the infrared image respectively. Expand represents the upsampling operation. The top layer L N = G N ;

[0021] S14. Fuse the Laplacian pyramids. Use the weighted average method to fuse the Laplacian pyramids of the RGB image and the infrared image layer by layer. The formula is as follows:

[0022]

[0023] Where, F K represents the K-th layer fused Laplacian pyramid image. α is the weight coefficient, which is adjusted according to specific requirements. Increase the weight of the infrared image when the light is insufficient.

[0024] S15. Reconstruct the fused image, and reconstruct the final fused image from the fused Laplacian pyramid. The formula is as follows:

[0025]

[0026] Among them, F represents the final fused image, and N represents the number of layers of the Laplacian pyramid.

[0027] Preferably, in S2, the eye fatigue feature data includes the aspect ratio of the eyes, the blinking speed, and the percentage of eye closure. The mouth fatigue feature data includes yawning, and the micro-expression fatigue feature data includes the movement of the depressor supercilii muscle and the corrugator supercilii muscle.

[0028] Preferably, the formula for the personalized threshold in S2 is as follows:

[0029]

[0030] Among them, T eye is the adaptive threshold of the aspect ratio of the eyes, EAR avg is the average aspect ratio of the eyes in the most recent 15 frames, EAR i is the aspect ratio of the eyes in the i-th frame, and k is the adjustment coefficient;

[0031] The formula for the adjustment coefficient k is as follows:

[0032]

[0033] Among them, k0 is the initial adjustment coefficient, β represents the sensitivity coefficient, μ represents the average value of the aspect ratio of the eyes in the most recent 15 frames, and σ represents the standard deviation of the aspect ratio of the eyes in the most recent 15 frames.

[0034] Preferably, in S3, the posture classification model is a deep learning model based on a convolutional neural network, and the classification is achieved by training the images of the head-down sleeping scenario.

[0035] Preferably, in S4, the fatigue parameter compensation is performed through the micro-expression comprehensive motion model. The specific formula is:

[0036]

[0037] F = W1 * f1 + W2 * f2;

[0038] Among them, f1 and f2 are the normalized features of the depressor supercilii muscle and the corrugator supercilii muscle respectively. Δy and Δx represent the vertical displacement difference of the depressor supercilii muscle and the horizontal convergence distance of the corrugator supercilii muscle respectively. Δy normal represents the reference vertical displacement in the normal state, Δx normal represents the reference horizontal distance in the normal state, and W1 and W2 are the corresponding weights.

[0039] Preferably, when face recognition is normal in S5, the formula of the comprehensive fatigue model A is:

[0040]

[0041] When face recognition is restricted, the fatigue parameter model B is:

[0042] P = w t ·P t ;

[0043] where w is the weight coefficient, P is the fatigue feature parameter, e represents the aspect ratio of the eyes, m is the aspect ratio of the mouth, w1 is the micro-expression, z is the blinking speed, b is the percentage of eye closure, and t is the special scenario of bowing the head and falling asleep.

[0044] Therefore, the present invention adopts the above-mentioned adaptive fatigue state detection method based on multi-modal feature fusion, and has the following beneficial effects:

[0045] (1) Collect face data through infrared and RGB cameras and fuse them, retain the detailed information and overall structure of the image, avoid the influence in low-light environments, then extract the feature information of the mouth, eyes, and micro-expressions, and then set personalized thresholds to exclude the influence of individual differences on the acquisition of fatigue features;

[0046] (2) When face recognition is restricted, judge whether it belongs to the scenario of bowing the head and falling asleep through pre-trained samples of bowing the head and falling asleep, avoid the judgment distortion caused by the lack of face features, and improve the detection accuracy in special cases;

[0047] (3) When face recognition is normal, enter the calculation of the comprehensive fatigue probability. When the mouth is blocked, consider the micro-expression fatigue feature for fatigue parameter compensation, avoid the decrease in detection accuracy caused by the lack of face features, and improve the stability of detection.

[0048] The technical solution of the present invention will be further described in detail below through the drawings and embodiments. Description of the Drawings

[0049] Figure 1 It is a block diagram of an adaptive fatigue state detection method based on multi-modal feature fusion of the present invention;

[0050] Figure 2 It is a model diagram of 468 feature points of a human face in an embodiment of an adaptive fatigue state detection method based on multi-modal feature fusion of the present invention;

[0051] Figure 3 It is an enlarged view of the right eye area in an embodiment of an adaptive fatigue state detection method based on multi-modal feature fusion of the present invention;

[0052] Figure 4Enlarged view of the left eye region in the embodiment of an adaptive fatigue state detection method based on multi-modal feature fusion according to the present invention;

[0053] Figure 5 Enlarged view of the mouth region in the embodiment of an adaptive fatigue state detection method based on multi-modal feature fusion according to the present invention;

[0054] Figure 6 Flowchart of an adaptive fatigue state detection method based on multi-modal feature fusion according to the present invention. Detailed implementation

[0055] The following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0056] Embodiment

[0057] As Figure 1 shown, the present invention provides an adaptive fatigue state detection method based on multi-modal feature fusion, including the following steps:

[0058] S1. Use an infrared camera and an RGB camera to synchronously collect facial images, and perform image fusion through the Laplacian pyramid method.

[0059] S11. In a night or low-light environment, the images collected by a single camera have large noise and serious loss of details, making it difficult to extract effective fatigue features, and the accuracy of fatigue detection is greatly reduced. Therefore, in this embodiment, an RGB image I RGB and an infrared image I IR are obtained by synchronously collecting facial images through an infrared camera and an RGB camera, and the infrared image is aligned with the RGB image through geometric transformations: rotation and scaling, and adjusted to a unified resolution.

[0060] S12. Gaussian pyramids are respectively constructed for the infrared image and the RGB image to obtain their respective Gaussian pyramid sequences. The number of layers N of the Gaussian pyramid is determined according to the image size and requirements, and the formula is as follows:

[0061]

[0062] Among them, and respectively represent the Kth layer Gaussian pyramid images of the RGB image and the infrared image, and Reduce is the Gaussian blur and downsampling operation;

[0063] S13. Construct a Laplacian pyramid, and construct Laplacian pyramids for the Gaussian pyramids of the RGB image and the infrared image respectively. The formula is as follows:

[0064]

[0065] Among them, and respectively represent the k-th layer Laplacian pyramid images of the RGB image and the infrared image. Expand represents the previous sampling operation. The top layer L N = G N ;

[0066] S14. Fuse the Laplacian pyramids, and use the weighted average method to fuse the Laplacian pyramids of the RGB image and the infrared image layer by layer. The formula is as follows:

[0067]

[0068] Among them, F K represents the k-th layer fused Laplacian pyramid image. α is the weight coefficient, which is adjusted according to specific requirements. When the light is insufficient, the weight of the infrared image is increased;

[0069] S15. Reconstruct the fused image, and reconstruct the final fused image from the fused Laplacian pyramid. The formula is as follows:

[0070]

[0071] Among them, F represents the final fused image, and N represents the number of layers of the Laplacian pyramid.

[0072] After fusing the facial images by the Laplacian pyramid method, the detailed information (high-frequency part) and the overall structure (low-frequency part) of the images can be retained, and at the same time, the influence of the low-light environment can be avoided, improving the accuracy of fatigue detection.

[0073] S2. Extract the fatigue feature data of the eyes, mouth and micro-expressions from the fused facial images, set personalized thresholds, and obtain fatigue features. The eye fatigue feature data includes the aspect ratio of the eyes, the blinking speed and the percentage of eye closure. The mouth fatigue feature data includes yawning. The micro-expression fatigue feature data includes the movement of the depressor supercilii muscle and the movement of the corrugator supercilii muscle.

[0074] During the facial recognition process, in this example, the MediaPipe Face Detection module is used for facial recognition, which can accurately, quickly and stably detect human faces and frame the human faces; as Figure 2As shown, in the detection of key facial feature points, MediaPipe Face Mesh is used to obtain facial mesh feature points. Face Mesh obtains 3D facial information through machine learning and uses the facial coordinate positions on the screen to estimate the transformation of the face in space. A total of 468 key 3D facial mesh points are recognized. In this example, the key points of the eyes, mouth, and eyebrows are mainly used.

[0075] Among the 468 key facial feature points, the key points of the eyes correspond to the key points of the right eye (33, 159, 158, 133, 145, 153) as Figure 3 shown, and the key points of the left eye (362, 385, 386, 263, 374, 380) as Figure 4 shown, and the key points of the mouth (78, 80, 310, 308, 318, 88) as Figure 5 shown.

[0076] The aspect ratio of the eyes is the average of the aspect ratios of the left and right eyes. That is, the formula for the aspect ratio of the eyes is where E 均 represents the average of the aspect ratios of the left and right eyes.

[0077] In the design of the aspect ratio threshold of the eyes, the eye shapes of each person are different. Some people have larger eye contours, while others have smaller eye contours. Judging whether it belongs to the closed-eye feature based on a single aspect ratio threshold has a large error; therefore, in this example, personalized thresholds are set according to individual feature differences to judge whether the eyes are closed.

[0078] The formula for the personalized threshold is as follows:

[0079]

[0080] where, T eye is the adaptive threshold of the aspect ratio of the eyes, EAR avg is the average of the aspect ratios of the eyes in the last 15 frames, EAR i is the aspect ratio value of the i-th frame, and k is the adjustment coefficient.

[0081] The formula for the adjustment coefficient k is as follows:

[0082]

[0083] where, k0 is the initial adjustment coefficient (usually taken as 1.5 - 2.0), β represents the sensitivity coefficient, μ represents the average of the aspect ratio values of the eyes in the last 15 frames, and σ represents the standard deviation of the aspect ratio values of the eyes in the last 15 frames. Finally, it is judged whether the collected eye data belongs to the blinking waveform according to the personalized threshold.

[0084] Under normal circumstances, the duration of each blink is very short, usually between 100 and 400 milliseconds. When a person is in a fatigued state, the duration of each blink will increase significantly, and sometimes there will even be a "microsleep" phenomenon, that is, the eyes are closed for a time exceeding the normal blink time, which may reach several seconds, showing an obvious difference from the blink speed in the normal state. Therefore, in this example, the blink speed is used in fatigue state detection.

[0085] The percentage of eye closure is the duration of eye closure within a unit of time. PERCLOS usually has three measurement methods: P70, P80, and Em. Among them, P80 has the best correlation with the degree of fatigue. P80 is the time ratio when the area where the eyelid covers the eyeball exceeds 80%. Therefore, the threshold adopted in this embodiment is that the initial distance (X I ) between the upper and lower eyelids in the natural state is used as the standard. When the real-time distance between the upper and lower eyelids is lower than 0.2X I , it is counted as eye closure. Through frame-by-frame recognition and judgment, it is judged whether the distance between the upper and lower eyelids exceeds the preset ratio. Finally, the number of frames determined to be fatigued and the total number of frames in a period of time are counted. The number of fatigued frames divided by the total number of frames is the percentage of eye closure.

[0086] When a person is drowsy, the characteristic of yawning will appear. During the process of yawning, the aspect ratio of the mouth will change periodically, and the interference characteristics such as speaking cause different ratio changes from those caused by yawning. Therefore, by judging the change of the aspect ratio of the mouth over time, it is judged whether the characteristic of yawning appears, and the fatigue state at this time is analyzed. The formula for calculating the aspect ratio of the mouth is M represents the aspect ratio value of the mouth, and P i is the key facial feature point.

[0087] S3. When it is impossible to extract facial feature data under the condition of limited facial recognition, through the analysis of the state of human fatigue, in the state of extreme fatigue, the situation of lowering the head and falling asleep will occur, resulting in the inability to recognize facial fatigue data. Therefore, the special scenario recognition of lowering the head and falling asleep is added. When the face is blocked or the recognition is limited, the pose classification model is used to recognize whether it is a scenario of lowering the head and falling asleep, and the deep learning model is used to recognize the user's pose of lowering the head and falling asleep. By training the images of the scenario of lowering the head and falling asleep, the convolutional neural network (CNN) is used to classify the images to judge whether it is a scenario of lowering the head and falling asleep; if it is determined to be a scenario of lowering the head and falling asleep, the feature of lowering the head and falling asleep is input into the S5 fatigue state judgment module as the basis for fatigue detection.

[0088] S4. In the state of mouth occlusion, consider the micro-expression fatigue feature for fatigue parameter compensation.

[0089] People often cover their mouths when yawning. This is a common habitual action. This mouth-covering action can interfere with the method of fatigue detection through mouth features. Since the mouth area is blocked, the system cannot accurately extract mouth features, reducing the accuracy of fatigue detection.

[0090] Through a large number of observations and in combination with the Facial Action Coding System, it is found that during the process of yawning, micro-expression features such as the eyebrows moving upward will occur. At this time, micro-expression features: depressor supercilii muscle movement and corrugator muscle movement are introduced for fatigue parameter compensation. When yawning, the depressor supercilii muscle will contract, causing the inner part of the eyebrows to move downward, and the key point (the brow tip) of the inner part of the eyebrows will move closer to the middle; when yawning, the corrugator muscle will contract, causing the skin at the root of the nose to wrinkle, and the key point at the root of the nose will move downward. According to the motion quantification of the depressor supercilii muscle movement and the corrugator muscle movement, this example provides a micro-expression comprehensive motion feature index.

[0091] The specific formula of the micro-expression comprehensive motion model is:

[0092]

[0093] F = W1 * f1 + W2 * f2;

[0094] Among them, f1 and f2 are the normalized features of the depressor supercilii muscle and the corrugator muscle respectively, Δy and Δx represent the vertical displacement difference of the depressor supercilii muscle and the horizontal convergence distance of the corrugator muscle respectively, and Δy normal represents the reference vertical displacement in the normal state, and Δx normal represents the reference horizontal distance in the normal state, and W1 and W2 are the corresponding weights.

[0095] S5. Construct a comprehensive fatigue model that fuses multiple fatigue features to judge the fatigue state. The fatigue state is divided into awake, fatigued, and severely fatigued.

[0096] When facial recognition is normal, the formula of the comprehensive fatigue model A is:

[0097]

[0098] Among them, w is the weight coefficient, P is the fatigue feature parameter, e represents the aspect ratio of the eyes, m is the aspect ratio of the mouth, w1 is the micro-expression, z is the blinking speed, and b is the percentage of eye closure; each weight coefficient is adjusted in real time according to the situation. When the mouth feature is blocked, w m is increased to 0 to compensate for the lack of yawning features. Similarly, when the mouth feature is not blocked, it is 0.

[0099] When facial recognition is restricted, the fatigue parameter model B is:

[0100] P = wt ·P t ;

[0101] Among them, t is a special scenario of bowing the head and sleeping; since the special scenario lacks facial fatigue features and the special scenario is bowing the head and sleeping, its weight coefficient is set to w t to 1.

[0102] S6. When a fatigue or severe fatigue state is detected, a fatigue warning message is reminded.

[0103] After detecting the fatigue state, timely and effective reminder is crucial for ensuring safety (such as driving safety) and improving work efficiency. In this example, the bracelet vibration method is used for fatigue reminder. Instructions are sent to the bracelet via Bluetooth to control the vibration motor (such as LRA linear resonant actuator) for vibration reminder.

[0104] Such as Figure 6 shown, a fatigue state detection method provided by this example defaults that the face can be recognized in the normal state before entering the program. First, facial data is collected by infrared and RGB cameras and fused to avoid the influence of low-light environments. Then, the lip, eye, and micro-expression feature information is extracted. Then, personalized thresholds are set to exclude the influence of individual differences on fatigue feature classification. When facial recognition is restricted, the special scenario (bowing the head and sleeping) recognition program is entered. It is judged whether it belongs to the bowing the head and sleeping scenario through pre-trained bowing the head and sleeping samples. If it is a special scenario, the fatigue state judgment program is entered. If it is not a special scenario, it re-enters the facial recognition restriction judgment. When there is no restriction in facial recognition, the comprehensive fatigue probability calculation is entered. When the lip is blocked, the micro-expression fatigue feature is considered for fatigue parameter compensation, and the fatigue state is judged through multi-fatigue feature fusion. If the fatigue state is fatigue or severe fatigue, a warning message is reminded.

[0105] Among them, the fusion of infrared and RGB images can retain the detailed information (high-frequency part) and overall structure (low-frequency part) of the image, while avoiding the influence of low-light environments and improving the accuracy of the fatigue detection method. During the process of fatigue recognition, special scenario analysis is carried out in the facial recognition restricted scenario. When the lip is blocked, fatigue parameter compensation is performed through the analysis of micro-expression features, which can improve the accuracy of fatigue detection to a certain extent. This example has a good adaptation to individual characteristics and the wakefulness with lack of facial feature information.

[0106] Therefore, the present invention adopts the above-mentioned adaptive fatigue state detection method based on multi-modal feature fusion. Through multi-modal feature fusion, it effectively retains image details, adapts to low-light environments, compensates for the lack of facial features, and combines personalized thresholds and special scenario judgment, significantly improving the accuracy and stability of fatigue state detection.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions of the present invention or make equivalent replacements, and these modifications or equivalent replacements do not enable the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An adaptive fatigue state detection method based on multi-modal feature fusion, characterized in that It includes the following steps: S1. Use an infrared camera and an RGB camera to synchronously collect facial images, and perform image fusion through the Laplacian pyramid method; S2. Extract fatigue feature data of the eyes, mouth, and micro-expressions from the fused facial images, set personalized thresholds, and obtain fatigue features; S3. When the face is occluded or recognition is limited, use a pose classification model to identify whether it is a scenario of bowing the head and sleeping. If it is determined to be a scenario of bowing the head and sleeping, input the bowing head and sleeping features into the S5 fatigue state judgment module as the basis for fatigue detection; S4. Under the state of mouth occlusion, consider micro-expression fatigue features for fatigue parameter compensation; S5. Construct a comprehensive fatigue model that fuses multiple fatigue features to judge the fatigue state. The fatigue states are divided into awake, fatigued, and severely fatigued; S6. When a fatigued or severely fatigued state is detected, give a fatigue warning message reminder.

2. The adaptive fatigue state detection method based on multi-modal feature fusion according to claim 1, wherein, The specific steps of S1 are as follows: S11. Synchronously collect facial images through an infrared camera and an RGB camera to obtain an RGB image I RGB and an infrared image I IR . Through geometric transformations: rotation and scaling, align the infrared image with the RGB image and adjust them to a unified resolution; S12. Build Gaussian pyramids for the infrared image and the RGB image respectively to obtain their respective Gaussian pyramid sequences. The number of layers N of the Gaussian pyramid is determined according to the image size and requirements. The formula is as follows: Among them, and represent the K-th layer Gaussian pyramid images of the RGB image and the infrared image respectively, and Reduce is the Gaussian blur and downsampling operation; S13. Build Laplacian pyramids. Build Laplacian pyramids for the Gaussian pyramids of the RGB image and the infrared image respectively. The formula is as follows: Among them, and respectively represent the K-th layer Laplacian pyramid images of the RGB image and the infrared image. Expand represents the last sampling operation, and the highest layer L N = G N ; S14. Fuse the Laplacian pyramids. Use the weighted average method to fuse the Laplacian pyramids of the RGB image and the infrared image layer by layer. The formula is as follows: Among them, F K represents the Laplacian pyramid image after fusion at the Kth layer, and α is the weight coefficient; S15. Reconstruct the fused image. Reconstruct the final fused image from the fused Laplacian pyramid. The formula is as follows: Where, F represents the final fused image, and N represents the number of layers of the Laplacian pyramid.

3. The adaptive fatigue state detection method based on multi-modal feature fusion according to claim 1, wherein: In S2, the eye fatigue feature data includes the aspect ratio of the eyes, the blinking speed, and the percentage of eyes closed. The mouth fatigue feature data includes yawning. The micro-expression fatigue feature data includes the movement of the depressor supercilii muscle and the movement of the corrugator supercilii muscle.

4. An adaptive fatigue state detection method based on multi-modal feature fusion according to claim 3, characterized in that, The formula for the personalized threshold in S2 is as follows: Among them, T eye is the adaptive threshold of the eye aspect ratio, EAR avg is the average value of the eye aspect ratio in the most recent 15 frames, EAR i is the eye aspect ratio value of the i-th frame, and k is the adjustment coefficient; The formula for the adjustment coefficient k is as follows: Where, k0 is the initial adjustment coefficient, β represents the sensitivity coefficient, μ represents the average value of the aspect ratio of the eyes in the last 15 frames, and σ represents the standard deviation of the aspect ratio of the eyes in the last 15 frames.

5. The adaptive fatigue state detection method based on multi-modal feature fusion according to claim 1, wherein: In S3, the pose classification model is a deep learning model based on a convolutional neural network, and classification is achieved by training images of the scenario of bowing the head and sleeping.

6. The adaptive fatigue state detection method based on multi-modal feature fusion according to claim 1, wherein: In S4, a micro-expression comprehensive movement model is used for fatigue parameter compensation. The specific formula is: F = W1*f1 + W2*f2; Among them, f1 and f2 are the normalized features of the procerus muscle and the corrugator supercilii muscle respectively, Δy and Δx represent the vertical displacement difference of the procerus muscle and the horizontal convergence distance of the corrugator supercilii muscle respectively, and Δy normal represents the reference vertical displacement in the normal state, and Δx normal represents the reference horizontal distance in the normal state, and W1 and W2 are the corresponding weights.

7. An adaptive fatigue state detection method based on multi-modal feature fusion according to claim 1, characterized in that When face recognition is normal in S5, the formula for the comprehensive fatigue model A is: When face recognition is limited, the fatigue parameter model B is: P = w t ·P t ; Where, w is the weight coefficient, P is the fatigue feature parameter, e represents the aspect ratio of the eyes, m is the aspect ratio of the mouth, w1 is the micro-expression, z is the blinking speed, b is the percentage of eyes closed, and t is the special scenario of bowing the head and sleeping.

Images