Driver fatigue state detection and early warning method based on multi-source data perception optimization

Abstract

The invention provides a driver fatigue state detection and early warning method based on multi-source data perception optimization. The method comprises the following steps: acquiring visual data, handheld data and physiological perception data generated in a vehicle driving process; performing feature extraction on the visual data, the hand-held data and the physiological perception data through an improved algorithm to obtain data features; carrying out feature fusion on the visual data features, the handheld data features and the physiological perception data features through a feature splicing and attention weighting mechanism; adjusting the Bayesian network model according to the driving data; inputting the visual data features, the handheld data features and the physiological perception data features which are subjected to feature fusion into a Bayesian network model, and generating a fatigue state level of the driver through the Bayesian network model; and performing driving early warning according to the fatigue state. According to the method, the limitation of driver fatigue state detection of a single sensor in a complex scene can be effectively overcome, and the probability of false alarm and missing alarm is reduced.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a method for detecting and warning of driver fatigue based on multi-source data perception optimization. Background Technology

[0002] With the continuous improvement of automotive intelligence and the increasing emphasis on road traffic safety across society, driver fatigue detection has become one of the core functions of advanced driver assistance systems. Statistics show that fatigued driving is one of the main causes of serious traffic accidents. Therefore, developing an accurate, real-time, and robust fatigue detection and early warning technology is of paramount practical significance for protecting people's lives and property and promoting the development of intelligent transportation systems.

[0003] In existing technologies, driver fatigue detection methods can be broadly categorized into two types: analysis methods based on a single data source and methods based on multi-sensor fusion. Early research and commercial systems largely relied on single-view analysis, capturing driver facial images through cameras and using computer vision algorithms to calculate indicators such as eye closure time percentage, blink frequency, and head nodding frequency. However, these methods are highly susceptible to interference from changes in ambient lighting, driver wearing glasses or masks, and head posture shifts. Their performance significantly degrades in complex lighting conditions such as at night and at tunnel entrances / exits, resulting in a high risk of false alarms and false negatives. To improve reliability, some technologies have introduced contact-based physiological sensors, such as monitoring heart rate variability through electrodes embedded in the steering wheel or seat. While these methods provide objective physiological indicators, they require contact with the driver's skin, potentially causing discomfort. Furthermore, their signals are easily contaminated by noise during vehicle movement or driver activity, leading to poor long-term stability and user acceptance.

[0004] In recent years, with the advancement of sensor technology, non-contact millimeter-wave radar has been used to detect drivers' subtle bodily movements, providing a new data dimension for fatigue detection. Meanwhile, multi-sensor information fusion has become an important development direction, aiming to combine the advantages of different modal data. However, existing fusion solutions mostly remain at the level of simple data or decision-making layer concatenation, failing to delve into the deep complementary relationships between different data sources and lacking the ability to adaptively adjust to dynamic driving scenarios. Furthermore, most algorithm models are complex and computationally intensive, making it difficult to meet the stringent constraints of real-time performance, power consumption, and cost on automotive embedded platforms, thus limiting their application prospects in large-scale pre-installed mass production. Summary of the Invention

[0005] This invention provides a driver fatigue state detection and early warning method based on multi-source data perception optimization, which can effectively overcome the limitations of single sensor in detecting driver fatigue state in complex scenarios such as drastic changes in lighting, drivers wearing glasses or masks, and large body movements, and reduce the probability of false alarms and missed alarms.

[0006] The first aspect of this invention provides a method for detecting and warning of driver fatigue based on multi-source data perception optimization, comprising the following steps: Acquire visual data, hand grip data, and physiological perception data generated during vehicle driving, and preprocess the visual data, hand grip data, and physiological perception data; Visual data features are obtained by using an improved lightweight YOLO algorithm; hand-held data features are obtained by using an MLP algorithm; and physiological perception data features are obtained by using a lightweight temporal convolutional network. Feature fusion is performed on visual data features, hand-held data features, and physiological perception data features through feature splicing and attention weighting mechanisms; Acquire driving data generated during vehicle driving and adjust the Bayesian network model based on the driving data; The visual data features, hand grip data features, and physiological perception data features that have been fused are input into the Bayesian network model, and the driver's fatigue level is generated through the Bayesian network model. Driver warnings based on fatigue levels.

[0007] Furthermore, the step of extracting features from visual data using the improved lightweight YOLO algorithm to obtain visual data features includes the following steps: Visual data is input into the improved YOLOv7-tiny model; The improved YOLOv7-tiny model outputs the coordinates of the driver's facial bounding box, eyelids, and corners of the mouth by setting a coordinate attention module and adding a shallow detection head; Based on the output coordinates, calculate the eye aspect ratio sequence and mouth aspect ratio sequence, and calculate the percentage of eye closure time within the set time window; A sliding window statistical feature extraction was performed on the eye aspect ratio sequence, mouth aspect ratio sequence, and eye closure time percentage value to construct a visual data feature vector.

[0008] Furthermore, the step of extracting features from the hand-held data using the MLP algorithm to obtain hand-held data features includes the following steps: Statistical features are calculated on the data held within a set time window to generate statistical features; the statistical features include the percentage of total holding time, the frequency of holding events, and the longest single holding duration. The calculated statistical features are combined and standardized to generate an initial behavioral feature vector. The initial behavioral feature vector is input into a lightweight multilayer perceptron, and the hand-holding data feature vector is output through the nonlinear transformation of the multilayer perceptron.

[0009] Furthermore, the step of extracting features from physiological perception data using a lightweight temporal convolutional network to obtain physiological perception data features includes the following steps: Set up a millimeter-wave radar to acquire physiological sensing data; process the echo signal from the millimeter-wave radar to separate the respiratory waveform signal and the heartbeat waveform signal; The respiratory waveform signal and the heartbeat waveform signal are segmented and normalized to generate input samples; The input samples are fed into a lightweight temporal convolutional network; the temporal convolutional network extracts the temporal patterns of the signals through dilated causal convolutional layers and residual convolutional modules, and outputs physiological perception data feature vectors.

[0010] Furthermore, the feature fusion of visual data features, hand-held data features, and physiological perception data features through feature splicing and attention weighting mechanisms includes the following steps: Visual data feature vectors, hand-grip data feature vectors, and physiological perception data feature vectors are concatenated to generate a multimodal joint feature vector; The multimodal joint feature vector is input into the attention network to generate and output an attention weight vector that is relevant to the current context. The multimodal joint feature vector is weighted and calibrated based on the attention weight vector to obtain a weighted fusion feature vector.

[0011] Furthermore, the acquisition of driving data generated during the vehicle driving process, and the adjustment of the Bayesian network model based on the driving data, includes the following steps: Obtain the continuous driving time and road condition information of the vehicle; The continuous driving time and road condition information are input into the adjustment function to calculate and adjust the prior probability distribution of fatigue state nodes in the Bayesian network model.

[0012] Furthermore, the process of inputting the fused visual data features, hand-grip data features, and physiological perception data features into a Bayesian network model to generate the driver's fatigue level includes the following steps: Input the multimodal joint features into the Bayesian network model with adjusted prior probabilities; By performing probabilistic inference through a Bayesian network model, the posterior probability of each fatigue level of a fatigue state node is calculated. According to the preset threshold rules, the fatigue level with the highest posterior probability or exceeding the threshold is determined as the driver's current fatigue level.

[0013] The second aspect of the present invention provides a driver fatigue state detection and early warning system based on multi-source data perception optimization, including a first processing module for acquiring visual data, hand grip data and physiological perception data generated during vehicle driving, and preprocessing the visual data, hand grip data and physiological perception data; The second processing module is used to extract features from visual data using an improved lightweight YOLO algorithm to obtain visual data features; to extract features from hand-held data using an MLP algorithm to obtain hand-held data features; and to extract features from physiological perception data using a lightweight temporal convolutional network to obtain physiological perception data features. The third processing module is used to perform feature fusion on visual data features, hand-held data features, and physiological perception data features through feature splicing and attention weighting mechanisms. The fourth processing module is used to acquire driving data generated during the vehicle driving process and adjust the Bayesian network model based on the driving data; The fifth processing module is used to input the visual data features, hand grip data features, and physiological perception data features that have been fused into the Bayesian network model, and generate the driver's fatigue level through the Bayesian network model. The sixth processing module is used to provide driving warnings based on fatigue status.

[0014] A third aspect of the present invention provides a computer device, comprising: Memory, transceiver, processor, and bus system; The memory is used to store programs; The processor is used to execute the program in the memory, including executing the driver fatigue state detection and early warning method based on multi-source data perception optimization described above; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

[0015] A fourth aspect of the present invention provides a readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the driver fatigue state detection and early warning method based on multi-source data perception optimization described above.

[0016] As can be seen from the above technical solutions, the present invention has the following advantages: This invention improves the overall accuracy of fatigue state detection by fusing three heterogeneous data sources: vision, hand-held operation, and millimeter-wave radar physiological perception. It utilizes an improved lightweight YOLO algorithm, MLP, and temporal convolutional networks for parallel feature extraction. The visual channel directly captures facial expressions and eye states, hand-held data indirectly reflects focus and control intention, while physiological perception data provides objective evidence of vital signs despite occlusion. This multi-dimensional, cross-validation approach effectively overcomes the limitations of single sensors in complex scenarios such as drastic changes in lighting, drivers wearing glasses or masks, and large body movements, reducing the probability of false alarms and false negatives.

[0017] Secondly, the algorithms used in this invention have all undergone lightweight design and optimization, especially the improved YOLOv7-tiny model and the lightweight temporal convolutional network, which significantly reduce computational complexity and memory consumption while ensuring high detection accuracy. This enables the entire algorithm to be efficiently deployed on automotive edge computing platforms such as TDA4VM, meeting the stringent requirements of the vehicle environment for real-time performance, power consumption, and reliability, and achieving a leap from theoretical models to practical automotive applications.

[0018] Finally, this invention introduces an adaptive learning mechanism based on driving context, which dynamically adjusts the prior probabilities of the Bayesian network model according to real-time acquired continuous driving duration and road condition complexity information. This makes the system's warning threshold no longer fixed, but can simulate human experience, becoming more sensitive as driving time increases and road conditions become more monotonous. This achieves personalized and precise warning timing, effectively avoiding the problem of untimely warnings caused by individual differences or different driving stages.

[0019] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from an examination of the following, or may be learned from the practice of the invention. Attached Figure Description

[0020] Figure 1 The method flowchart provided by the present invention. Detailed Implementation

[0021] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] Example 1 The implementation method in this embodiment can be implemented in a system, on a server, or on a terminal; no specific limitation is made. The method in this application will be described from the perspective of system implementation below. As shown in the figure, a driver fatigue state detection and early warning method based on multi-source data perception optimization includes the following steps: Acquire visual data, hand grip data, and physiological perception data generated during vehicle driving, and preprocess the visual data, hand grip data, and physiological perception data; Visual data features are obtained by using an improved lightweight YOLO algorithm; hand-held data features are obtained by using an MLP algorithm; and physiological perception data features are obtained by using a lightweight temporal convolutional network. Feature fusion is performed on visual data features, hand-held data features, and physiological perception data features through feature splicing and attention weighting mechanisms; Acquire driving data generated during vehicle driving and adjust the Bayesian network model based on the driving data; The visual data features, hand grip data features, and physiological perception data features that have been fused are input into the Bayesian network model, and the driver's fatigue level is generated through the Bayesian network model. Driver warnings based on fatigue levels.

[0023] Visual data is captured by in-vehicle cameras to observe the driver's facial expressions and eye movements; hand grip data is acquired through capacitive sensors embedded in the steering wheel to monitor the driver's hand grip; and physiological data is collected non-contactly by millimeter-wave radar to obtain vital signs such as heart rate and respiratory rate. Before being input into the core algorithm, this raw data undergoes its own preprocessing pipeline for standardization to eliminate noise and unify its format.

[0024] For visual data, the improved lightweight YOLO algorithm performs real-time inference on each frame of the image, quickly locating the driver's face and identifying the coordinates of key feature points such as eyelids and corners of the mouth. It then calculates quantitative fatigue index sequences such as eye aspect ratio, mouth aspect ratio, and PERCLOS. For hand-grip data, the system performs statistical analysis within a time window on the hand-grip / hand-off state sequences generated by capacitive sensors, extracting behavioral features such as hand-off duration and frequency. These features are then abstracted at a higher level using a lightweight multilayer perceptron to obtain a feature vector representing driving focus. For physiological perception data, breathing and heartbeat waveforms separated from millimeter-wave radar echo signals are input into a lightweight temporal convolutional network. This network utilizes dilated causal convolution structures to extract deep temporal patterns and rhythmic features of physiological signals.

[0025] After feature extraction, the system proceeds to the feature fusion stage. Feature vectors from the three approaches are concatenated into a joint multimodal feature vector. This joint vector is then fed into an attention-weighted module, which automatically learns and assigns importance weights to different feature dimensions based on the current context information. This enhances key information and suppresses redundant or noisy information, ultimately generating an optimized fused feature vector. Simultaneously, the system acquires continuous driving time and vehicle dynamic data reflecting road condition complexity in real time via the vehicle bus. Based on this driving context information, a predefined adjustment function dynamically corrects the prior probability distribution of fatigue state nodes in the Bayesian network, making the model's basic judgments more closely aligned with the actual driving load.

[0026] Finally, the system inputs the attention-weighted fused feature vector as observational evidence into the Bayesian network model, which has undergone prior adjustment. The Bayesian network performs probabilistic inference to calculate the posterior probability of the driver being at each fatigue level. Based on preset decision rules, such as selecting the level corresponding to the highest posterior probability, the system determines the driver's real-time fatigue level and triggers matching multi-level warning signals.

[0027] Example 2 The difference between this embodiment and Embodiment 1 is that the step of extracting features from visual data using the improved lightweight YOLO algorithm to obtain visual data features includes the following steps: Visual data is input into the improved YOLOv7-tiny model; The improved YOLOv7-tiny model outputs the coordinates of the driver's facial bounding box, eyelids, and corners of the mouth by setting a coordinate attention module and adding a shallow detection head; Based on the output coordinates, calculate the eye aspect ratio sequence and mouth aspect ratio sequence, and calculate the percentage of eye closure time within the set time window; A sliding window statistical feature extraction was performed on the eye aspect ratio sequence, mouth aspect ratio sequence, and eye closure time percentage value to construct a visual data feature vector.

[0028] The improved lightweight YOLOv7-tiny model incorporates an attention module that integrates coordinates into the neck network. This attention module adaptively emphasizes fatigue-related features in both channel and spatial dimensions. A shallow detection head specifically designed for small object detection is also added to enhance sensitivity to minute areas such as the eyes and mouth. The model first outputs a bounding box of the driver's face, then precisely locates the key pixel coordinates of the eyelid contours and mouth corners within the box. Based on these coordinates, the system calculates the eye aspect ratio and mouth aspect ratio in real time for each frame, resulting in two dynamically changing temporal sequences. Simultaneously, within a set sliding time window, the system calculates the percentage of frames with eye aspect ratios below the eye-closing threshold, thus determining the percentage of time the eyes are closed. Finally, statistical features are extracted from these continuous aspect ratio sequences and PERCLOS values ​​within the sliding window, calculating features such as mean, variance, and peak frequency. These features collectively form a comprehensive feature vector representing the visual fatigue state within the current time period, completing the transformation from the original image to structured digital features.

[0029] Example 3 The difference between this embodiment and embodiment two is that the step of extracting features from the hand-held data using the MLP algorithm to obtain hand-held data features includes the following steps: Statistical features are calculated on the data held within a set time window to generate statistical features; the statistical features include the percentage of total holding time, the frequency of holding events, and the longest single holding duration. The calculated statistical features are combined and standardized to generate an initial behavioral feature vector. The initial behavioral feature vector is input into a lightweight multilayer perceptron, and the hand-holding data feature vector is output through the nonlinear transformation of the multilayer perceptron.

[0030] The system acquires hand-grip data via a capacitive sensor mounted on the steering wheel. During feature extraction, primary feature engineering is performed on the state sequence within a sliding time window synchronized with visual analysis, calculating a set of statistical features with clear behavioral significance. These features primarily include: the proportion of total hand-free time to the window duration, used to quantify the overall degree of hand detachment; the frequency of hand-free events per unit time, used to assess the frequency of attentional distraction; and the longest duration of a single consecutive hand-free event, used to capture potentially severe periods of distraction or fatigue. Subsequently, these statistical features with different dimensions are combined into an initial feature vector and standardized to eliminate differences in magnitude. This vector is then fed into a lightweight multilayer perceptron. The multilayer perceptron, through the nonlinear activation functions of its hidden layers, learns the complex combination relationships and higher-order interactions between the aforementioned basic statistical features, automatically uncovering more discriminative deep behavioral patterns. The output of the multilayer perceptron is the final hand-grip data feature vector.

[0031] Example 4 The difference between this embodiment and embodiment three is that the step of extracting features from physiological perception data using a lightweight temporal convolutional network to obtain physiological perception data features includes the following steps: Set up a millimeter-wave radar to acquire physiological sensing data; process the echo signal from the millimeter-wave radar to separate the respiratory waveform signal and the heartbeat waveform signal; The respiratory waveform signal and the heartbeat waveform signal are segmented and normalized to generate input samples; The input samples are fed into a lightweight temporal convolutional network; the temporal convolutional network extracts the temporal patterns of the signals through dilated causal convolutional layers and residual convolutional modules, and outputs physiological perception data feature vectors.

[0032] The frequency-modulated continuous wave emitted by the millimeter-wave radar is reflected by the driver's chest cavity, generating a mid-frequency echo signal containing micro-motion information caused by breathing and heartbeat. By performing a distance-dimensional Fast Fourier Transform on the echo signal, a specific distance gate corresponding to the distance to the driver's chest cavity is obtained, and the temporal changes in the phase or amplitude of the signal at this distance gate are extracted. Subsequently, using digital filters with different passbands, the lower-frequency respiratory waveform signal and the higher-frequency heartbeat waveform signal are separated from this one-dimensional temporal signal. To further meet the model input requirements, the system segments these two types of waveform signals according to fixed durations and performs normalization operations to generate a batch of regular input samples. These samples are fed in parallel into a lightweight temporal convolutional network. The core of the temporal convolutional network is stacked dilated causal convolutional layers and residual modules; dilated convolution can achieve a large receptive range with a small number of layers, effectively capturing the long-period dependence of respiratory rhythm, while causal convolution ensures that the processing conforms to temporal logic. Residual connections ensure the effective flow of gradients in the deep network, improving training stability. The network eventually converges from the last layer and outputs a high-dimensional feature vector, which constitutes the internal physiological perception data features for the system to judge fatigue state.

[0033] Example 5 The difference between this embodiment and embodiment four is that the feature fusion of visual data features, hand-held data features, and physiological perception data features through feature splicing and attention weighting mechanisms includes the following steps: Visual data feature vectors, hand-grip data feature vectors, and physiological perception data feature vectors are concatenated to generate a multimodal joint feature vector; The multimodal joint feature vector is input into the attention network to generate and output an attention weight vector that is relevant to the current context. The multimodal joint feature vector is weighted and calibrated based on the attention weight vector to obtain a weighted fusion feature vector.

[0034] The feature concatenation operation involves concatenating the three sub-vectors, which are distributed differently in the feature space, end-to-end along a dimensional axis to form a unified multimodal joint feature vector. This multimodal joint feature vector is then input into an attention network module, which integrates instantaneous contextual information to generate an attention weight vector corresponding to the dimension of the input vector. This module typically consists of fully connected layers and activation functions, and through learning, it can automatically analyze the internal structure of the current joint features. Each scalar value in this weight vector represents the importance of the corresponding dimensional feature in the current decision. Subsequently, the system performs weight calibration, multiplying the joint feature vector element-wise with the attention weight vector to enhance the strength of key evidence features while weakening features that may contain noise or are weakly correlated with the current state. The output after this process is a context-aware optimized fused feature vector. This mechanism enables the system to flexibly handle complex situations such as sudden changes in lighting causing temporary visual feature failure, or intense driving causing physiological signals to be interfered with by motion, dynamically shifting the decision focus towards the most reliable data source.

[0035] Example 6 The difference between this embodiment and embodiment five is that the step of acquiring driving data generated during the vehicle driving process and adjusting the Bayesian network model based on the driving data includes the following steps: Obtain the continuous driving time and road condition information of the vehicle; The continuous driving time and road condition information are input into the adjustment function to calculate and adjust the prior probability distribution of fatigue state nodes in the Bayesian network model.

[0036] The system can acquire driving data in real time via the vehicle's CAN bus or various onboard sensors. This data includes two main components: first, the accumulated continuous driving time since the start of the current trip or the last sufficient rest; and second, information reflecting road condition complexity, which can be derived by analyzing multi-source information such as real-time vehicle speed stability, steering wheel angle change entropy, or navigation map data. These two data points are input into a predefined adjustment function, which maps continuous driving time to an incremental contribution to the baseline fatigue probability, while simultaneously quantifying road condition complexity as an adjustment coefficient. For example, on straight, unobstructed highways, this coefficient amplifies the fatigue accumulation effect, while in complex urban road conditions requiring frequent maneuvering, the coefficient is appropriately reduced. The output of the adjustment function is directly used to update the prior probability distribution of the top-level hidden node representing the fatigue state in the Bayesian network. That is, given the same observational evidence, such as the same yawning frequency or heart rate, the system's initial assumption about a driver just starting to drive is different from that of a driver who has been driving continuously for several hours, indicating that the driver is in a fatigued state.

[0037] Example 7 The difference between this embodiment and Embodiment Six lies in the following steps: inputting the visual data features, hand-grip data features, and physiological perception data features obtained through feature fusion into a Bayesian network model, and generating the driver's fatigue level through the Bayesian network model: Input the multimodal joint features into the Bayesian network model with adjusted prior probabilities; By performing probabilistic inference through a Bayesian network model, the posterior probability of each fatigue level of a fatigue state node is calculated. According to the preset threshold rules, the fatigue level with the highest posterior probability or exceeding the threshold is determined as the driver's current fatigue level.

[0038] The attention-weighted fused feature vectors are input into the Bayesian network model as a set of observed deterministic evidence. The top-level nodes representing fatigue states and the child nodes representing various types of evidence are connected via conditional probability tables, encoding the probabilistic causal relationships between different features and fatigue levels. Before inference begins, the fatigue state nodes in the network are assigned dynamically adjusted prior probability distributions based on the current continuous driving time and road condition complexity, forming the initial empirical benchmark for the system's judgment.

[0039] Subsequently, the Bayesian network initiates conditional probability calculation and probability propagation based on the fused feature evidence input. By combining the combined effects of all evidence, it calculates the posterior probability of the driver actually being in each of the following levels of fatigue: "awake," "mildly fatigued," "moderately fatigued," or "severely fatigued," given all the available known information.

[0040] Finally, the system interprets and maps the obtained posterior probability distribution according to preset decision rules. For example, the maximum a posteriori probability criterion is used to take the level with the highest probability value as the final output; or a grading threshold is set so that the highest level alarm is triggered only when the posterior probability of severe fatigue exceeds a certain high threshold (such as 80%).

[0041] Example 8 A driver fatigue state detection and early warning system based on multi-source data perception optimization includes a first processing module for acquiring visual data, hand grip data and physiological perception data generated during vehicle driving, and preprocessing the visual data, hand grip data and physiological perception data. The second processing module is used to extract features from visual data using an improved lightweight YOLO algorithm to obtain visual data features; to extract features from hand-held data using an MLP algorithm to obtain hand-held data features; and to extract features from physiological perception data using a lightweight temporal convolutional network to obtain physiological perception data features. The third processing module is used to perform feature fusion on visual data features, hand-held data features, and physiological perception data features through feature splicing and attention weighting mechanisms. The fourth processing module is used to acquire driving data generated during the vehicle driving process and adjust the Bayesian network model based on the driving data; The fifth processing module is used to input the visual data features, hand grip data features, and physiological perception data features that have been fused into the Bayesian network model, and generate the driver's fatigue level through the Bayesian network model. The sixth processing module is used to provide driving warnings based on fatigue status.

[0042] Example 9 A computer device, comprising: Memory, transceiver, processor, and bus system; The memory is used to store programs; The processor is used to execute the program in the memory, including executing the driver fatigue state detection and early warning method based on multi-source data perception optimization described above; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

[0043] Example 10 A readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the driver fatigue state detection and early warning method based on multi-source data perception optimization described above.

[0044] In summary, this invention improves the overall accuracy of fatigue state detection by fusing three heterogeneous data sources: vision, hand-held operation, and millimeter-wave radar physiological perception. It utilizes an improved lightweight YOLO algorithm, MLP, and temporal convolutional networks for parallel feature extraction. The visual channel directly captures facial expressions and eye states, hand-held data indirectly reflects focus and control intention, while physiological perception data provides objective evidence of vital signs despite occlusion. This multi-dimensional, cross-validation approach effectively overcomes the limitations of single sensors in complex scenarios such as drastic changes in lighting, drivers wearing glasses or masks, and large body movements, reducing the probability of false alarms and false negatives.

[0045] Secondly, the algorithms used in this invention have all undergone lightweight design and optimization, especially the improved YOLOv7-tiny model and the lightweight temporal convolutional network, which significantly reduce computational complexity and memory consumption while ensuring high detection accuracy. This enables the entire algorithm to be efficiently deployed on automotive edge computing platforms such as TDA4VM, meeting the stringent requirements of the vehicle environment for real-time performance, power consumption, and reliability, and achieving a leap from theoretical models to practical automotive applications.

[0046] Finally, this invention introduces an adaptive learning mechanism based on driving context, which dynamically adjusts the prior probabilities of the Bayesian network model according to real-time acquired continuous driving duration and road condition complexity information. This makes the system's warning threshold no longer fixed, but can simulate human experience, becoming more sensitive as driving time increases and road conditions become more monotonous. This achieves personalized and precise warning timing, effectively avoiding the problem of untimely warnings caused by individual differences or different driving stages.

[0047] Those skilled in the art will recognize that the units of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the invention.

[0048] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical functional division. In actual implementation, there may be other division methods, such as multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored.

[0049] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0050] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0051] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting and warning of driver fatigue based on multi-source data perception optimization, characterized in that, Includes the following steps: Acquire visual data, hand grip data, and physiological perception data generated during vehicle driving, and preprocess the visual data, hand grip data, and physiological perception data; Visual data features are obtained by using an improved lightweight YOLO algorithm; hand-held data features are obtained by using an MLP algorithm; and physiological perception data features are obtained by using a lightweight temporal convolutional network. Feature fusion is performed on visual data features, hand-held data features, and physiological perception data features through feature splicing and attention weighting mechanisms; Acquire driving data generated during vehicle driving and adjust the Bayesian network model based on the driving data; The visual data features, hand grip data features, and physiological perception data features that have been fused are input into the Bayesian network model, and the driver's fatigue level is generated through the Bayesian network model. Driver warnings based on fatigue levels.

2. The method for driver fatigue state detection and early warning based on multi-source data perception optimization according to claim 1, characterized in that, The step of extracting features from visual data using an improved lightweight YOLO algorithm to obtain visual data features includes the following steps: Visual data is input into the improved YOLOv7-tiny model; The improved YOLOv7-tiny model outputs the coordinates of the driver's facial bounding box, eyelids, and corners of the mouth by setting a coordinate attention module and adding a shallow detection head; Based on the output coordinates, calculate the eye aspect ratio sequence and mouth aspect ratio sequence, and calculate the percentage of eye closure time within the set time window; A sliding window statistical feature extraction was performed on the eye aspect ratio sequence, mouth aspect ratio sequence, and eye closure time percentage value to construct a visual data feature vector.

3. The method for driver fatigue state detection and early warning based on multi-source data perception optimization according to claim 1, characterized in that, The step of extracting features from hand-held data using the MLP algorithm to obtain hand-held data features includes the following steps: Statistical features are calculated on the data held within a set time window to generate statistical features; the statistical features include the percentage of total holding time, the frequency of holding events, and the longest single holding duration. The calculated statistical features are combined and standardized to generate an initial behavioral feature vector. The initial behavioral feature vector is input into a lightweight multilayer perceptron, and the hand-holding data feature vector is output through the nonlinear transformation of the multilayer perceptron.

4. The method for driver fatigue state detection and early warning based on multi-source data perception optimization according to claim 1, characterized in that, The step of extracting features from physiological perception data using a lightweight temporal convolutional network to obtain physiological perception data features includes the following steps: Set up a millimeter-wave radar to acquire physiological sensing data; process the echo signal from the millimeter-wave radar to separate the respiratory waveform signal and the heartbeat waveform signal; The respiratory waveform signal and the heartbeat waveform signal are segmented and normalized to generate input samples; The input samples are fed into a lightweight temporal convolutional network; the temporal convolutional network extracts the temporal patterns of the signals through dilated causal convolutional layers and residual convolutional modules, and outputs physiological perception data feature vectors.

5. The method for driver fatigue state detection and early warning based on multi-source data perception optimization according to claim 1, characterized in that, The feature fusion of visual data features, hand-held data features, and physiological perception data features through feature splicing and attention weighting mechanisms includes the following steps: Visual data feature vectors, hand-grip data feature vectors, and physiological perception data feature vectors are concatenated to generate a multimodal joint feature vector; The multimodal joint feature vector is input into the attention network to generate and output an attention weight vector that is relevant to the current context. The multimodal joint feature vector is weighted and calibrated based on the attention weight vector to obtain a weighted fusion feature vector.

6. The method for driver fatigue state detection and early warning based on multi-source data perception optimization according to claim 1, characterized in that, The process of acquiring driving data generated during vehicle driving and adjusting the Bayesian network model based on the driving data includes the following steps: Obtain the continuous driving time and road condition information of the vehicle; The continuous driving time and road condition information are input into the adjustment function to calculate and adjust the prior probability distribution of fatigue state nodes in the Bayesian network model.

7. The method for driver fatigue state detection and early warning based on multi-source data perception optimization according to claim 1, characterized in that, The process of inputting the fused visual data features, hand grip data features, and physiological perception data features into a Bayesian network model, and generating the driver's fatigue level through the Bayesian network model, includes the following steps: Input the multimodal joint features into the Bayesian network model with adjusted prior probabilities; By performing probabilistic inference through a Bayesian network model, the posterior probability of each fatigue level of a fatigue state node is calculated. According to the preset threshold rules, the fatigue level with the highest posterior probability or exceeding the threshold is determined as the driver's current fatigue level.

8. A driver fatigue state detection and early warning system based on multi-source data perception optimization, characterized in that, It includes a first processing module, which is used to acquire visual data, hand grip data and physiological perception data generated during the vehicle driving process, and to preprocess the visual data, hand grip data and physiological perception data; The second processing module is used to extract features from visual data using an improved lightweight YOLO algorithm to obtain visual data features; to extract features from hand-held data using an MLP algorithm to obtain hand-held data features; and to extract features from physiological perception data using a lightweight temporal convolutional network to obtain physiological perception data features. The third processing module is used to perform feature fusion on visual data features, hand-held data features, and physiological perception data features through feature splicing and attention weighting mechanisms. The fourth processing module is used to acquire driving data generated during the vehicle driving process and adjust the Bayesian network model based on the driving data; The fifth processing module is used to input the visual data features, hand grip data features, and physiological perception data features that have been fused into the Bayesian network model, and generate the driver's fatigue level through the Bayesian network model. The sixth processing module is used to provide driving warnings based on fatigue status.

9. A computer device, characterized in that, include: Memory, transceiver, processor, and bus system; The memory is used to store programs; The processor is used to execute the program in the memory, including executing the driver fatigue state detection and early warning method based on multi-source data perception optimization as described in any one of claims 1 to 7; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

10. A readable storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by the processor, they implement the steps of the driver fatigue state detection and early warning method based on multi-source data perception optimization as described in any one of claims 1 to 7.

Images