Fatigue detection method and system based on visual-imu feature-level fusion
By using a vision-IMU feature-level fusion method, the problems of environmental factors and individual differences in driver fatigue detection are solved, and accurate fatigue detection is achieved in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAIBEI NORMAL UNIVERSITY
- Filing Date
- 2026-01-26
- Publication Date
- 2026-06-09
AI Technical Summary
Existing driver fatigue detection technologies are greatly affected by environmental factors, resulting in decreased accuracy in facial key point positioning and an inability to adapt to individual differences among drivers, leading to misjudgments or missed detections.
A visual-IMU feature-level fusion method is adopted. Through synchronous acquisition and preprocessing of multi-source heterogeneous data, a cascaded visual perception pipeline and a deep temporal modeling network are constructed. Combined with adaptive Kalman filtering and a two-layer LSTM network, personalized fatigue state determination is achieved.
It improves the accuracy and robustness of key point positioning in complex environments, reduces false alarms and missed alarms, and achieves personalized and accurate fatigue detection.
Smart Images
Figure CN122176672A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of fatigue detection technology, and in particular to a fatigue detection method and system based on visual-IMU feature-level fusion. Background Technology
[0002] With the rapid development of the intelligent transportation industry, driving safety has become a core issue of concern to the whole society, and driver fatigue is one of the main causes of traffic accidents.
[0003] Real-time and accurate fatigue detection technology can issue timely warnings to drivers, effectively reducing the accident rate. Therefore, it has important application value in commercial vehicles, passenger vehicles and intelligent driving assistance systems, and the research and optimization of related technologies has become a hot topic in the industry.
[0004] Currently, mainstream driver fatigue detection technologies all have shortcomings. Pure vision detection technology uses cameras to collect facial images and extracts features such as the aspect ratio of the eyes and the opening ratio of the mouth to determine fatigue status. However, it is greatly affected by environmental factors: strong direct sunlight, low light environment, and facial occlusion encountered during driving can lead to a significant decrease in the accuracy of facial key point localization, which in turn causes feature extraction errors. At the same time, traditional vision models lack targeted enhancement for fatigue-sensitive areas, resulting in insufficient efficiency and accuracy in key point detection. Moreover, most of them use fixed thresholds to determine fatigue status, which cannot adapt to individual differences such as blinking frequency and facial contours of different drivers, and are prone to misjudgment or missed judgment.
[0005] Therefore, fatigue detection methods and systems based on visual-IMU feature-level fusion are needed. Summary of the Invention
[0006] The purpose of this invention is to solve the problems pointed out in the background art, and to propose a fatigue detection method and system based on visual-IMU feature-level fusion.
[0007] To achieve the above objectives, the present invention adopts the following technical solution: a fatigue detection method and system based on visual-IMU feature-level fusion, comprising the following steps:
[0008] S1: Synchronous Acquisition and Preprocessing of Multi-Source Heterogeneous Data: Using a low-power sensing module that communicates with the edge computing unit, the driver's facial video stream and IMU six-axis inertial data (including three-axis acceleration and three-axis angular velocity) are acquired synchronously; timestamp alignment technology is used to ensure strict synchronization of visual frames and inertial sampling points on the time axis, and adaptive Kalman filtering is used to denoise the raw IMU signal, providing a high signal-to-noise ratio data benchmark for subsequent feature fusion;
[0009] S2: Constructing a cascaded visual perception pipeline based on a coarse-to-fine strategy: First, the lightweight detection network RTMDet is used to extract spatial features of image frames and generate candidate facial bounding boxes, achieving robust localization of facial regions under complex lighting conditions; then, the localized facial regions are cropped and normalized to a preset resolution, and input into a customized MobileNetV3-Small regression network integrating an inverse residual structure and an SE attention module to perform fine-grained facial keypoint coordinate regression, extracting a set of 20 keypoint coordinates containing core features of the periorbital and lip areas; finally, based on the coordinate set, the eye aspect ratio (EAR) reflecting the eye opening state and the mouth opening ratio (MAR) reflecting the mouth opening dimension are calculated.
[0010] S3: Perform head attitude calculation based on adaptive Kalman filtering: Establish a discrete state space model, and perform fusion filtering on the original triaxial acceleration and triaxial angular velocity data of the IMU by setting the dynamic process noise covariance Q and the observation noise covariance R to eliminate sensor noise and temperature drift interference; then, use the filtered vector data to calculate the pitch, roll and yaw angles that characterize the spatial features of head motion, and generate a continuous attitude angle time series that can reflect the evolution trend of the driver's head attitude;
[0011] S4: Perform time-scale alignment and multimodal feature-level fusion of heterogeneous data: Use linear interpolation algorithms to resample visual features (EAR values, MAR values) and inertial features (attitude angle sequences) with different sampling frequencies to ensure that heterogeneous data are aligned under the same clock reference; then perform normalization processing on the aligned data to eliminate dimensional differences, and construct a high-dimensional fused feature vector containing facial micro-expression morphology and head motion dynamics through feature concatenation, providing semantically complete input for subsequent deep temporal modeling;
[0012] S5: Construct a deep temporal modeling network for fatigue evolution analysis: Input the multimodal fusion feature vector into a preset two-layer long short-term memory network (LSTM), use its internal gating mechanism to extract the nonlinear dependence of facial morphology and head movements in the time dimension, and output the fatigue prediction probability value representing the current driver state in real time through mapping of fully connected layers and activation functions.
[0013] S6: Fatigue state determination based on personalized temporal modeling: The constructed multimodal temporal feature sequence is input into a two-layer LSTM network. The network gating mechanism is used to perform deep temporal modeling on the driving behavior habits of a specific user (including unique eye-closing patterns and head posture changes). The individual fatigue evolution pattern of the user is analyzed and the predicted probability is output. Combined with the dual constraints of preset probability threshold and duration, fatigue state is determined only when the predicted probability continuously exceeds the limit within the set duration. In this way, while achieving personalized and accurate detection, false alarms caused by physiological blinking or brief head movements are effectively filtered out.
[0014] S7: Perform model optimization and deployment for edge computing platforms: Perform operator fusion and INT8 quantization on the optimized model, deploy it to an embedded edge computing platform (such as Jetson Nano), and use the hardware acceleration engine to realize real-time inference output of the multimodal detection scheme.
[0015] Preferably, step S2 specifically includes:
[0016] S201: Constructing a cascaded visual perception pipeline for hierarchical feature extraction: First, the lightweight detection network RTMDet is called to perform spatial semantic analysis on the original image frames. Through its multi-scale feature fusion mechanism, the face target is locked, and candidate bounding boxes with robustness to sudden changes in illumination are generated. Then, the face image within the bounding box is extracted and normalized resampling is performed. It is then input into a customized MobileNetV3-Small network that integrates a self-attention mechanism (SE module) and lightweight inverted residuals. The nonlinear feature mapping capability of this network is used to achieve high-dimensional regression and real-time localization of the core facial key points.
[0017] S202: Perform feature dimension compression based on facial geometric constraints: Using the coordinates of 20 core key points output by the MobileNetV3-Small network, extract key points around the eyes according to a preset index mapping relationship; Transform the high-dimensional spatial coordinate vector into a one-dimensional temporal scalar representing the degree of eyelid closure by calculating the eye aspect ratio (EAR), thereby achieving standardized extraction of blinking action features;
[0018] S203: Perform quantitative characterization of mouth morphology features: Select key points of the corners of the mouth and the edges of the upper and lower lips in the coordinate set, and obtain the mouth opening ratio (MAR) by calculating the Euclidean distance ratio of the vertical opening distance to the horizontal width of the mouth; This index is used to map complex mouth deformation into numerical features, providing a feature basis for subsequent decoupling of speech and yawning actions by combining inertial data;
[0019] S204: Calculation of the moving average of the eye aspect ratio (EAR) based on a sliding window. Combined with attenuation coefficient With lower threshold Generate dynamic threshold TH Its formula is: TH .
[0020] Preferably, step S3 specifically comprises:
[0021] S301: Acquire raw IMU data via the MPU6050 sensor, with the sampling frequency set to 100Hz;
[0022] S302: Based on triaxial acceleration components , , The initial pitch and roll values are calculated using the following formulas:
[0023] , ;
[0024] S303: Perform adaptive Kalman filtering based on state-space model: Construct discrete state-space equations containing attitude angles and drift errors, and perform weighted fusion of accelerometer and gyroscope data in the prediction and update phases by dynamically setting the process noise covariance Q and observation noise covariance R.
[0025] S304: Generate a drift-resistant steady-state attitude angle sequence: suppress sensor random noise and zero-point drift, and output a filtered sequence containing continuous pitch, roll and yaw angles to characterize the spatial dynamic trajectory of the driver's head.
[0026] Preferably, step S4 specifically comprises:
[0027] S401: Perform multimodal heterogeneous feature reconstruction: Real-time retrieval of eye aspect ratio (EAR) and mouth opening ratio (MAR) generated by the visual perception pipeline as facial micro-expression features, and synchronous retrieval of the filtered posture angle sequence generated by the posture calculation module as head motion dynamic features to construct a two-dimensional feature space.
[0028] S402: Perform cross-modal time scale alignment and frequency compensation: To address the asymmetry between visual frame rate and inertial sampling frequency, a linear interpolation algorithm is used to resample visual features and inertial features, aligning asynchronous data streams to a unified time step benchmark, ensuring strict correspondence between facial morphological features and spatial motion features in the temporal sequence.
[0029] S403: Perform normalization processing on visual feature data and inertial feature data to eliminate dimensional differences;
[0030] S404: This method fuses two types of data through feature concatenation to generate a multimodal fusion feature vector. The formula is as follows: concat ,in, For visual feature vectors, This is the inertial eigenvector.
[0031] Preferably, step S5 specifically comprises:
[0032] S501: Configure a dual-layer LSTM network, with each layer containing 128 hidden units;
[0033] S502: Input multimodal fusion feature vector, set sequence length to 30, update unit state and hidden state through input gate, forget gate and output gate;
[0034] S503: The hidden state at time step 1 is input into the fully connected layer, and the fatigue state prediction probability is output through the Softmax function. The formula is as follows: Softmax ;
[0035] S504: The training process uses the cross-entropy loss function, the formula of which is: Optimize network parameters.
[0036] Preferably, step S6 specifically comprises:
[0037] S601: Input the multimodal temporal feature sequence into a two-layer LSTM network, and use the memory units and gating mechanism inside the network to perform temporal modeling on the current user's driving behavior habits and fatigue evolution patterns, and output the fatigue prediction probability representing the current state.
[0038] S602: Preset the probability threshold and duration threshold for fatigue determination, wherein the probability threshold ranges from 0.5 to 0.9 and the duration threshold ranges from 1 to 5 seconds;
[0039] S603: Real-time monitoring and comparison of the fatigue prediction probability with the probability threshold;
[0040] S604: The driver is determined to be fatigued only when the fatigue prediction probability continuously exceeds the probability threshold within the duration threshold, thereby filtering out false alarms caused by non-fatigue physiological blinking or brief head movements.
[0041] A fatigue detection system based on visual-IMU feature-level fusion, applicable to the aforementioned fatigue detection method based on visual-IMU feature-level fusion, is characterized by comprising:
[0042] The visual data acquisition module acquires video streams of the driving process and loads them as image frames;
[0043] The IMU data acquisition module collects triaxial acceleration and triaxial angular velocity data.
[0044] The visual feature extraction module is characterized by comprising a cascaded face detection submodule and a key point regression submodule. This module first locates the face region in the image and generates candidate bounding boxes using the lightweight detection network RTMDet, leveraging the network's efficient spatial feature extraction capabilities to handle complex lighting interference. Subsequently, the face region image is cropped and input into a customized MobileNetV3-Small regression network to extract fine-grained facial key point coordinates, and the eye aspect ratio (EAR) and mouth opening ratio are calculated based on the extracted coordinate set.
[0045] The IMU data processing module performs Kalman filtering on the raw IMU data, calculates the head attitude angles Pitch, Roll, and Yaw values, and generates a filtered attitude angle sequence.
[0046] The feature fusion module performs temporal alignment on visual feature data and inertial feature data, performs normalization processing, and generates a multimodal fusion feature vector through feature concatenation.
[0047] The temporal modeling module inputs the multimodal fusion feature vector into a two-layer LSTM network to perform temporal modeling and outputs the fatigue state prediction probability.
[0048] The fatigue determination module compares the predicted probability of fatigue state with a preset threshold to determine the fatigue state.
[0049] The personalized calibration module collects multimodal data from specific users, fine-tunes the LSTM network, and generates a unique weight file.
[0050] The embedded deployment module performs format conversion and quantization optimization on the model and deploys it to the embedded platform.
[0051] Compared with the prior art, the advantages and positive effects of the present invention are as follows:
[0052] 1. Multimodal feature-level fusion significantly improves environmental adaptability. This invention innovatively integrates visual information (facial micro-expressions) with inertial measurement data (head posture dynamics) at the feature level, constructing a cross-modal fatigue-sensitive feature system. By introducing a coarse-to-fine cascaded detection and regression mechanism in the visual perception module, combined with the attention enhancement structure of a lightweight network, the accuracy and robustness of keypoint localization under non-ideal conditions such as complex lighting and partial facial occlusion are effectively improved. Compared with traditional pure vision solutions, this method significantly suppresses feature fluctuations caused by environmental interference through compensation and joint analysis of inertial data, thereby greatly improving the stability and reliability of the system in different driving scenarios while ensuring real-time detection.
[0053] 2. Dynamic Threshold Determination Mechanism Based on Personalized Temporal Modeling. This invention not only achieves the extraction and fusion of multimodal features, but also introduces a personalized modeling mechanism based on user behavior habits. By collecting visual-inertial temporal data of specific drivers and using a two-layer LSTM network for deep temporal pattern learning, the system can adaptively learn individual behavioral characteristics such as blink frequency and head movement amplitude, thereby generating personalized fatigue determination thresholds and duration constraints. This mechanism can effectively distinguish between fatigue states and normal physiological activities (such as brief eye closure, speaking, and natural head rotation), significantly reducing false alarms and missed alarms caused by individual differences and instantaneous behavior, and improving detection accuracy and user adaptability.
[0054] 3. Lightweight Deployment and Real-Time Inference Optimization for Embedded Platforms. At the model implementation level, this invention features system-level lightweight design and deployment optimization for edge computing environments. By pruning the feature extraction network structure, fusing operators, and performing INT8 quantization, the computational complexity and memory footprint of the model are significantly reduced, enabling high-frame-rate real-time inference on low-power embedded platforms such as Jetson Nano. This design not only ensures the real-time response capability of the fatigue detection system but also provides a feasible technical path for its practical application in resource-constrained environments such as in-vehicle terminals, further enhancing the engineering practicality and promotional value of this solution. Attached Figure Description
[0055] Figure 1 This is a flowchart illustrating the process of the method of the present invention;
[0056] Figure 2 This is a flowchart of the system workflow of the present invention. Detailed Implementation
[0057] To make the objectives, technical solutions, and specific implementations of this invention clearer, the technical solutions, constituent elements, and operational processes of this invention will be described in detail below with reference to the accompanying drawings. This section aims to provide specific and operable embodiments to help those skilled in the art understand and implement this invention.
[0058] First, it should be noted that some terms used in the description of this invention may have multiple forms of expression, but their meanings are essentially the same, as follows:
[0059] The expressions "for example," "exemplarily," and "for instance" appearing in this text are only used to explain and illustrate the relevant technical content through specific examples, and do not indicate that the listed situations have special priority or exclusivity. Any reasonable modifications or extensions based on the inventive concept should fall within the protection scope of this invention. The term "and / or" indicates that there may be a parallel, selective, or combined relationship between the two or more connected items, and its specific meaning depends on its technical context. In the textual description, the terms "image" and "picture" both refer to two-dimensional image data acquired by a visual sensor when their differences are not explicitly distinguished, and the two can be used interchangeably. The expressions "of," "corresponding," and "corresponding" all indicate that there is a correlation or mapping relationship between two objects when there is no special distinction in the context, and their meanings are considered equivalent.
[0060] When dealing with parameter or variable representations, subscript forms (such as...) ) and non-subscript form (such as The difference between the two is merely a matter of writing style; they refer to the same entity and should be considered as the same object in understanding and application.
[0061] To systematically clarify the technical problem to be solved by the present invention, the technical means adopted, and the beneficial effects achieved, the following will describe in detail the various components, data flow, processing steps, and coordination mechanism of the technical solution of the present invention, based on specific embodiments of the present invention and in conjunction with relevant illustrations.
[0062] Please see Figure 1 This invention provides a technical solution: a fatigue detection method and system based on visual-IMU feature-level fusion, comprising the following steps:
[0063] S1: Synchronous Acquisition and Preprocessing of Multi-Source Heterogeneous Data: Using a low-power sensing module that communicates with the edge computing unit, the driver's facial video stream and IMU six-axis inertial data (including three-axis acceleration and three-axis angular velocity) are acquired synchronously; timestamp alignment technology is used to ensure strict synchronization of visual frames and inertial sampling points on the time axis, and adaptive Kalman filtering is used to denoise the raw IMU signal, providing a high signal-to-noise ratio data benchmark for subsequent feature fusion;
[0064] S2: Constructing a cascaded visual perception pipeline based on a coarse-to-fine strategy: First, the lightweight detection network RTMDet is used to extract spatial features of image frames and generate candidate facial bounding boxes, achieving robust localization of facial regions under complex lighting conditions; then, the localized facial regions are cropped and normalized to a preset resolution, and input into a customized MobileNetV3-Small regression network integrating an inverse residual structure and an SE attention module to perform fine-grained facial keypoint coordinate regression, extracting a set of 20 keypoint coordinates containing core features of the periorbital and lip areas; finally, based on the coordinate set, the eye aspect ratio (EAR) reflecting the eye opening state and the mouth opening ratio (MAR) reflecting the mouth opening dimension are calculated.
[0065] S3: Perform head attitude calculation based on adaptive Kalman filtering: Establish a discrete state space model, and perform fusion filtering on the original triaxial acceleration and triaxial angular velocity data of the IMU by setting the dynamic process noise covariance Q and the observation noise covariance R to eliminate sensor noise and temperature drift interference; then, use the filtered vector data to calculate the pitch, roll and yaw angles that characterize the spatial features of head motion, and generate a continuous attitude angle time series that can reflect the evolution trend of the driver's head attitude;
[0066] S4: Perform time-scale alignment and multimodal feature-level fusion of heterogeneous data: Use linear interpolation algorithms to resample visual features (EAR values, MAR values) and inertial features (attitude angle sequences) with different sampling frequencies to ensure that heterogeneous data are aligned under the same clock reference; then perform normalization processing on the aligned data to eliminate dimensional differences, and construct a high-dimensional fused feature vector containing facial micro-expression morphology and head motion dynamics through feature concatenation, providing semantically complete input for subsequent deep temporal modeling;
[0067] S5: Construct a deep temporal modeling network for fatigue evolution analysis: Input the multimodal fusion feature vector into a preset two-layer long short-term memory network (LSTM), use its internal gating mechanism to extract the nonlinear dependence of facial morphology and head movements in the time dimension, and output the fatigue prediction probability value representing the current driver state in real time through mapping of fully connected layers and activation functions.
[0068] S6: Fatigue state determination based on personalized temporal modeling: The constructed multimodal temporal feature sequence is input into a two-layer LSTM network. The network gating mechanism is used to perform deep temporal modeling on the driving behavior habits of a specific user (including unique eye-closing patterns and head posture changes). The individual fatigue evolution pattern of the user is analyzed and the predicted probability is output. Combined with the dual constraints of preset probability threshold and duration, fatigue state is determined only when the predicted probability continuously exceeds the limit within the set duration. In this way, while achieving personalized and accurate detection, false alarms caused by physiological blinking or brief head movements are effectively filtered out.
[0069] S7: Perform model optimization and deployment for edge computing platforms: Perform operator fusion and INT8 quantization on the optimized model, deploy it to an embedded edge computing platform (such as Jetson Nano), and use the hardware acceleration engine to realize real-time inference output of the multimodal detection scheme.
[0070] Furthermore, step S2 above specifically includes:
[0071] S201: Constructing a cascaded visual perception pipeline for hierarchical feature extraction: First, the lightweight detection network RTMDet is called to perform spatial semantic analysis on the original image frames. Through its multi-scale feature fusion mechanism, the face target is locked, and candidate bounding boxes with robustness to sudden changes in illumination are generated. Then, the face image within the bounding box is extracted and normalized resampling is performed. It is then input into a customized MobileNetV3-Small network that integrates a self-attention mechanism (SE module) and lightweight inverted residuals. The nonlinear feature mapping capability of this network is used to achieve high-dimensional regression and real-time localization of the core facial key points.
[0072] S202: Perform feature dimension compression based on facial geometric constraints: Using the coordinates of 20 core key points output by the MobileNetV3-Small network, extract key points around the eyes according to a preset index mapping relationship; Transform the high-dimensional spatial coordinate vector into a one-dimensional temporal scalar representing the degree of eyelid closure by calculating the eye aspect ratio (EAR), thereby achieving standardized extraction of blinking action features;
[0073] S203: Perform quantitative characterization of mouth morphology features: Select key points of the corners of the mouth and the edges of the upper and lower lips in the coordinate set, and obtain the mouth opening ratio (MAR) by calculating the Euclidean distance ratio of the vertical opening distance to the horizontal width of the mouth; This index is used to map complex mouth deformation into numerical features, providing a feature basis for subsequent decoupling of speech and yawning actions by combining inertial data;
[0074] S204: Calculation of the moving average of the eye aspect ratio (EAR) based on a sliding window. Combined with attenuation coefficient With lower threshold Generate dynamic threshold TH Its formula is: TH .
[0075] Furthermore, in the above, step S3 specifically refers to:
[0076] S301: Acquire raw IMU data via the MPU6050 sensor, with the sampling frequency set to 100Hz;
[0077] S302: Based on triaxial acceleration components , , The initial pitch and roll values are calculated using the following formulas:
[0078] , ;
[0079] S303: Perform adaptive Kalman filtering based on state-space model: Construct discrete state-space equations containing attitude angles and drift errors, and perform weighted fusion of accelerometer and gyroscope data in the prediction and update phases by dynamically setting the process noise covariance Q and observation noise covariance R.
[0080] S304: Generate a drift-resistant steady-state attitude angle sequence: suppress sensor random noise and zero-point drift, and output a filtered sequence containing continuous pitch, roll and yaw angles to characterize the spatial dynamic trajectory of the driver's head.
[0081] Furthermore, in the above, step S4 specifically refers to:
[0082] S401: Perform multimodal heterogeneous feature reconstruction: Real-time retrieval of eye aspect ratio (EAR) and mouth opening ratio (MAR) generated by the visual perception pipeline as facial micro-expression features, and synchronous retrieval of the filtered posture angle sequence generated by the posture calculation module as head motion dynamic features to construct a two-dimensional feature space.
[0083] S402: Perform cross-modal time scale alignment and frequency compensation: To address the asymmetry between visual frame rate and inertial sampling frequency, a linear interpolation algorithm is used to resample visual features and inertial features, aligning asynchronous data streams to a unified time step benchmark, ensuring strict correspondence between facial morphological features and spatial motion features in the temporal sequence.
[0084] S403: Perform normalization processing on visual feature data and inertial feature data to eliminate dimensional differences;
[0085] S404: This method fuses two types of data through feature concatenation to generate a multimodal fusion feature vector. The formula is as follows: concat ,in, For visual feature vectors, This is the inertial eigenvector.
[0086] Furthermore, in the above, step S5 specifically refers to:
[0087] S501: Configure a dual-layer LSTM network, with each layer containing 128 hidden units;
[0088] S502: Input multimodal fusion feature vector, set sequence length to 30, update unit state and hidden state through input gate, forget gate and output gate;
[0089] S503: The hidden state at time step 1 is input into the fully connected layer, and the fatigue state prediction probability is output through the Softmax function. The formula is as follows: Softmax ;
[0090] S504: The training process uses the cross-entropy loss function, the formula of which is: Optimize network parameters.
[0091] Furthermore, in the above, step S6 specifically refers to:
[0092] S601: Input the multimodal temporal feature sequence into a two-layer LSTM network, and use the memory units and gating mechanism inside the network to perform temporal modeling on the current user's driving behavior habits and fatigue evolution patterns, and output the fatigue prediction probability representing the current state.
[0093] S602: Preset the probability threshold and duration threshold for fatigue determination, wherein the probability threshold ranges from 0.5 to 0.9 and the duration threshold ranges from 1 to 5 seconds;
[0094] S603: Real-time monitoring and comparison of the fatigue prediction probability with the probability threshold;
[0095] S604: The driver is determined to be fatigued only when the fatigue prediction probability continuously exceeds the probability threshold within the duration threshold, thereby filtering out false alarms caused by non-fatigue physiological blinking or brief head movements.
[0096] A fatigue detection system based on visual-IMU feature-level fusion, applicable to the aforementioned fatigue detection method based on visual-IMU feature-level fusion, includes:
[0097] Visual data acquisition module: Uses a Deli camera to capture video streams of the driving process, loads them as image frames, and ensures clear images of the facial area;
[0098] IMU data acquisition module: Employs MPU6050 sensor, which acquires triaxial acceleration and triaxial angular velocity data at a frequency of 100Hz via I²C protocol, providing high-frequency attitude dynamic information;
[0099] The visual feature extraction module includes a cascaded face detection submodule and a keypoint regression submodule. This module first uses the lightweight detection network RTMDet to locate the face region in real time from a global perspective and generate candidate bounding boxes. It leverages its spatial feature enhancement capabilities to improve robustness to complex lighting and occlusion environments. Subsequently, the face region image is cropped and input into a customized MobileNetV3-Small regression network. Its lightweight inverted residual structure is used to extract fine-grained facial keypoints, outputting a set of facial keypoint coordinates. Based on the coordinate set, the eye aspect ratio (EAR) and mouth opening ratio are calculated.
[0100] IMU data processing module: Performs Kalman filtering (Q=0.01, R=0.1) on the raw IMU data, calculates the head attitude angle Pitch value, Roll value, and Yaw value, and generates a smooth and time-consistent filtered attitude angle sequence;
[0101] Feature fusion module: Performs temporal alignment (unified timestamp), frequency compensation (linear interpolation), and normalization on visual feature data and inertial feature data, and generates a 35-dimensional multimodal fusion feature vector through feature concatenation;
[0102] Temporal modeling module: Input the multimodal fusion feature vector into a two-layer LSTM network (128 hidden units per layer) to perform temporal modeling, and output the fatigue state prediction probability through the Softmax function;
[0103] Personalized calibration module: Collects 5 minutes of multimodal data (awake and simulated fatigue states) from a specific user, fine-tunes the LSTM network, and generates a quantized, proprietary weight file;
[0104] Embedded deployment module: Performs ONNX format conversion and TensorRT quantization optimization on the model, deploys it to the NVIDIA Jetson Nano platform, and achieves low power consumption, high frame rate (29.1FPS) real-time inference to output detection results.
[0105] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A fatigue detection method based on visual-IMU feature-level fusion, characterized in that, Includes the following steps: S1: Synchronous Acquisition and Preprocessing of Multi-Source Heterogeneous Data: Using a low-power sensing module that communicates with the edge computing unit, the driver's facial video stream and IMU six-axis inertial data (including three-axis acceleration and three-axis angular velocity) are acquired synchronously; timestamp alignment technology is used to ensure strict synchronization of visual frames and inertial sampling points on the time axis, and adaptive Kalman filtering is used to denoise the raw IMU signal, providing a high signal-to-noise ratio data benchmark for subsequent feature fusion; S2: Construct a cascaded visual perception pipeline based on a coarse-to-fine strategy: First, use the lightweight detection network RTMDet to extract spatial features of image frames and generate candidate bounding boxes for faces to achieve robust localization of face regions under complex lighting conditions. Subsequently, the located facial region is cropped and normalized to a preset resolution, and input into a customized MobileNetV3-Small regression network integrating an inverse residual structure and an SE attention module. Fine-grained facial keypoint coordinate regression is performed to extract a set of 20 keypoint coordinates containing core features of the periorbital and lip areas. Finally, based on the coordinate set, the eye aspect ratio (EAR), which reflects the eye opening state, and the mouth opening ratio (MAR), which reflects the mouth opening dimension, are calculated. S3: Perform head attitude calculation based on adaptive Kalman filtering: Establish a discrete state space model, and perform fusion filtering on the original triaxial acceleration and triaxial angular velocity data of the IMU by setting the dynamic process noise covariance Q and the observation noise covariance R to eliminate sensor noise and temperature drift interference; then, use the filtered vector data to calculate the pitch, roll and yaw angles that characterize the spatial features of head motion, and generate a continuous attitude angle time series that can reflect the evolution trend of the driver's head attitude; S4: Perform time-scale alignment and multimodal feature-level fusion of heterogeneous data: Use linear interpolation algorithms to resample visual features (EAR values, MAR values) and inertial features (attitude angle sequences) with different sampling frequencies to ensure that heterogeneous data are aligned under the same clock reference; then perform normalization processing on the aligned data to eliminate dimensional differences, and construct a high-dimensional fused feature vector containing facial micro-expression morphology and head motion dynamics through feature concatenation, providing semantically complete input for subsequent deep temporal modeling; S5: Construct a deep temporal modeling network for fatigue evolution analysis: Input the multimodal fusion feature vector into a preset two-layer long short-term memory network (LSTM), use its internal gating mechanism to extract the nonlinear dependence of facial morphology and head movements in the time dimension, and output the fatigue prediction probability value representing the current driver state in real time through mapping of fully connected layers and activation functions. S6: Fatigue state determination based on personalized temporal modeling: The constructed multimodal temporal feature sequence is input into a two-layer LSTM network. The network gating mechanism is used to perform deep temporal modeling on the driving behavior habits of a specific user (including unique eye-closing patterns and head posture changes). The individual fatigue evolution pattern of the user is analyzed and the predicted probability is output. Combined with the dual constraints of preset probability threshold and duration, fatigue state is determined only when the predicted probability continuously exceeds the limit within the set duration. In this way, while achieving personalized and accurate detection, false alarms caused by physiological blinking or brief head movements are effectively filtered out. S7: Perform model optimization and deployment for edge computing platforms: Perform operator fusion and INT8 quantization on the optimized model, deploy it to an embedded edge computing platform (such as Jetson Nano), and use the hardware acceleration engine to realize real-time inference output of the multimodal detection scheme.
2. The fatigue detection method based on visual-IMU feature-level fusion according to claim 1, characterized in that, Step S2 specifically includes: S201: Constructing a cascaded visual perception pipeline for hierarchical feature extraction: First, the lightweight detection network RTMDet is called to perform spatial semantic analysis on the original image frames. Through its multi-scale feature fusion mechanism, the face target is locked, and candidate bounding boxes with robustness to sudden changes in illumination are generated. Then, the face image within the bounding box is extracted and normalized resampling is performed. It is then input into a customized MobileNetV3-Small network that integrates a self-attention mechanism (SE module) and lightweight inverted residuals. The nonlinear feature mapping capability of this network is used to achieve high-dimensional regression and real-time localization of the core facial key points. S202: Perform feature dimension compression based on facial geometric constraints: Using the coordinates of 20 core key points output by the MobileNetV3-Small network, extract key points around the eyes according to a preset index mapping relationship; Transform the high-dimensional spatial coordinate vector into a one-dimensional temporal scalar representing the degree of eyelid closure by calculating the eye aspect ratio (EAR), thereby achieving standardized extraction of blinking action features; S203: Perform quantitative characterization of mouth morphology features: Select key points of the corners of the mouth and the edges of the upper and lower lips in the coordinate set, and obtain the mouth opening ratio (MAR) by calculating the Euclidean distance ratio of the vertical opening distance to the horizontal width of the mouth; This index is used to map complex mouth deformation into numerical features, providing a feature basis for subsequent decoupling of speech and yawning actions by combining inertial data; S204: Calculation of the moving average of the eye aspect ratio (EAR) based on a sliding window. Combined with attenuation coefficient With lower threshold Generate dynamic threshold TH Its formula is: TH .
3. The fatigue detection method based on visual-IMU feature-level fusion according to claim 1, characterized in that: Step S3 specifically involves: S301: Acquire raw IMU data via the MPU6050 sensor, with the sampling frequency set to 100Hz; S302: Based on triaxial acceleration components , , The initial pitch and roll values are calculated using the following formulas: , ; S303: Perform adaptive Kalman filtering based on state-space model: Construct discrete state-space equations containing attitude angles and drift errors, and perform weighted fusion of accelerometer and gyroscope data in the prediction and update phases by dynamically setting the process noise covariance Q and observation noise covariance R. S304: Generate a drift-resistant steady-state attitude angle sequence: suppress sensor random noise and zero-point drift, and output a filtered sequence containing continuous pitch, roll and yaw angles to characterize the spatial dynamic trajectory of the driver's head.
4. The fatigue detection method based on visual-IMU feature-level fusion according to claim 1, characterized in that: Step S4 specifically involves: S401: Perform multimodal heterogeneous feature reconstruction: Real-time retrieval of eye aspect ratio (EAR) and mouth opening ratio (MAR) generated by the visual perception pipeline as facial micro-expression features, and synchronous retrieval of the filtered posture angle sequence generated by the posture calculation module as head motion dynamic features to construct a two-dimensional feature space. S402: Perform cross-modal time scale alignment and frequency compensation: To address the asymmetry between visual frame rate and inertial sampling frequency, a linear interpolation algorithm is used to resample visual features and inertial features, aligning asynchronous data streams to a unified time step benchmark, ensuring strict correspondence between facial morphological features and spatial motion features in the temporal sequence. S403: Perform normalization processing on visual feature data and inertial feature data to eliminate dimensional differences; S404: This method fuses two types of data through feature concatenation to generate a multimodal fusion feature vector. The formula is as follows: concat ,in, For visual feature vectors, This is the inertial eigenvector.
5. The fatigue detection method based on visual-IMU feature-level fusion according to claim 1, characterized in that: Step S5 specifically involves: S501: Configure a dual-layer LSTM network, with each layer containing 128 hidden units; S502: Input multimodal fusion feature vector, set sequence length to 30, update unit state and hidden state through input gate, forget gate and output gate; S503: The hidden state at time step 1 is input into the fully connected layer, and the fatigue state prediction probability is output through the Softmax function. The formula is as follows: Softmax ; S504: The training process uses the cross-entropy loss function, the formula of which is: Optimize network parameters.
6. The fatigue detection method based on visual-IMU feature-level fusion according to claim 1, characterized in that: Step S6 specifically includes: S601: Input the multimodal temporal feature sequence into a two-layer LSTM network, and use the memory units and gating mechanism inside the network to perform temporal modeling on the current user's driving behavior habits and fatigue evolution patterns, and output the fatigue prediction probability representing the current state. S602: Preset the probability threshold and duration threshold for fatigue determination, wherein the probability threshold ranges from 0.5 to 0.9 and the duration threshold ranges from 1 to 5 seconds; S603: Real-time monitoring and comparison of the fatigue prediction probability with the probability threshold; S604: The driver is determined to be fatigued only when the fatigue prediction probability continuously exceeds the probability threshold within the duration threshold, thereby filtering out false alarms caused by non-fatigue physiological blinking or brief head movements.
7. A fatigue detection system based on visual-IMU feature-level fusion, applicable to the fatigue detection method based on visual-IMU feature-level fusion as described in any one of claims 1-6, characterized in that, include: The visual data acquisition module acquires video streams of the driving process and loads them as image frames; The IMU data acquisition module collects triaxial acceleration and triaxial angular velocity data. The visual feature extraction module is characterized by including a cascaded face detection submodule and a key point regression submodule; the lightweight detection network RTMDet is used to locate the face region in the image in real time and generate the region of interest (ROI); then a customized MobileNetV3-Small network is used to extract the fine-grained facial key point coordinates of the ROI, and the eye aspect ratio EAR value and mouth opening ratio value are calculated based on the coordinate set. The IMU data processing module performs Kalman filtering on the raw IMU data, calculates the head attitude angles Pitch, Roll, and Yaw values, and generates a filtered attitude angle sequence. The feature fusion module performs temporal alignment on visual feature data and inertial feature data, performs normalization processing, and generates a multimodal fusion feature vector through feature concatenation. The temporal modeling module inputs the multimodal fusion feature vector into a two-layer LSTM network to perform temporal modeling and outputs the fatigue state prediction probability. The personalized calibration module collects multimodal data from specific users, trains an LSTM network, and generates a dedicated weight file. The embedded deployment module performs format conversion and quantization optimization on the model and deploys it to the embedded platform.