Electroencephalogram-course deflection angle multi-mode fusion fatigue driving detection method and system
By employing a multimodal fusion method combining EEG and heading angle, and utilizing Hilbert transform and support vector machine regression models, the problem of insufficient generalization ability in existing fatigue driving detection models is solved, achieving accurate identification of fatigue states and logically clear detection results.
Patent Information
- Application Number
- CN202512051152.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-31
- Publication Date
- 2026-02-27
AI Technical Summary
Existing fatigue driving detection methods rely excessively on internal physiological signals, failing to effectively capture functional degradation at the driving behavior level. Furthermore, they lack a quantitative coupling relationship between neural activity and external behavior, resulting in insufficient generalization ability and interpretability of the detection models.
A multimodal fusion method of EEG-heading angle is adopted. The instantaneous phase of EEG signal and heading angle signal is extracted by Hilbert transform, the phase difference is calculated and significance analysis is performed, and a support vector machine regression model is constructed to achieve accurate identification of fatigue state.
It achieves coupled analysis of physiological state and external behavior, improves the comprehensiveness and accuracy of detection, overcomes the limitations of single-modality detection, enhances the model's generalization ability and robustness, and provides logically clear detection results.
Smart Images
Figure CN121572990A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of fatigue detection technology, specifically relating to a multimodal fusion method and system for fatigue driving detection based on EEG and heading angle. Background Technology
[0002] Mental fatigue is a prevalent health problem in modern society, having a wide-ranging negative impact on individual cognitive function and behavioral performance. This is especially true during tasks requiring sustained attention, such as long-distance driving, where fatigue significantly increases the risk of accidents. Fatigue driving has become a major threat to road traffic safety, endangering not only the lives of drivers and passengers but also posing a serious hazard to other road users. With the continuous expansion of road traffic and the general increase in driving time, developing efficient and reliable fatigue driving monitoring and early warning technologies has significant social implications and application needs.
[0003] In recent years, physiological state recognition methods based on electroencephalography (EEG) signals have become a research hotspot in the fields of human factors engineering and neural engineering. By analyzing the time-domain, frequency-domain, and nonlinear characteristics of pure EEG signals, researchers are committed to constructing a mapping model from neural activity to behavioral performance to achieve objective identification of individual fatigue states. In the field of fatigue driving detection, existing technologies mainly include methods based on single EEG features and methods that fuse EEG with multiple physiological signals such as electrocardiogram (ECG). These methods have improved the recognition accuracy and system robustness to some extent, but still have significant limitations: on the one hand, they rely too heavily on internal physiological signals, making it difficult to effectively capture functional degradation at the behavioral level during driving; on the other hand, they lack quantitative modeling of the coupling relationship between neural activity and external behavior, resulting in insufficient generalization ability and interpretability of the model in real driving environments. Summary of the Invention
[0004] This invention provides a multimodal fusion method and system for detecting fatigued driving based on EEG and heading angle, in order to solve the technical problems in the prior art, such as over-reliance on internal physiological signals, inability to effectively capture functional degeneration at the driving behavior level, and lack of quantitative modeling of the coupling relationship between neural activity and external driving behavior, resulting in insufficient generalization ability and interpretability of the detection model.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal fusion method for detecting fatigue driving based on EEG and heading angle includes the following steps: Obtaining EEG signals from different subjects in normal and fatigue states. and heading angle signal ; Electroencephalogram (EEG) signals of different subjects in normal and fatigue states and heading angle signal The instantaneous phase is extracted by Hilbert transform, and the phase difference at each time point is calculated based on the instantaneous phase to obtain the phase lock value between the EEG signal of different EEG channels and the heading angle signal for each subject. Significance analysis was performed on the phase lock values of EEG signals and heading angle signals of different EEG channels for each subject to obtain the set of phase lock values of the target EEG channels for all subjects, and state labels were marked according to different states. A fatigue detection regression model was constructed, which was used to detect fatigue in subjects based on the set of phase-locked values of the target EEG channels of all subjects and their corresponding different state labels.
[0006] The acquisition of electroencephalogram (EEG) signals from different subjects in normal and fatigue states ( ) ) and heading angle signal ( Specifically, this involves using a PC and a driving simulator to simulate the driving process. A heading angle acquisition device is mounted on the driving simulator to collect heading angle signals during the driving process. The data collection involved different subjects wearing EEG caps while driving to collect brainwave signals. The collection.
[0007] The electroencephalogram (EEG) signals of different subjects in normal and fatigue states were analyzed. and heading angle signal Before extracting the instantaneous phase using Hilbert transform, it is necessary to analyze the EEG signals of different subjects in normal and fatigue states. and heading angle signal Preprocessing is performed, specifically: processing the EEG signals. Electrode localization, bandpass filtering, independent component analysis, and rereference were performed on the preprocessed EEG signals. As the pure EEG signal ultimately used; for the heading angle signal Bandpass filtering and data interpolation are performed on the preprocessed heading angle signal. The final pure heading angle signal (HDA) is used.
[0008] Hilbert transforms were performed on the pure EEG signal and the pure HAD signal to obtain their respective analytical signals. Based on these analytical signals, the instantaneous phases of the EEG and HAD signals were extracted. Based on these instantaneous phases, the phase difference between the EEG and HAD signals at each time point was calculated for different EEG channels. The phase differences were then converted into complex numbers. Finally, the average of these complex phase differences at each time point was calculated within a time window to obtain the phase lock value for each subject's EEG signal and HAD signal across different EEG channels. The formula for calculating the phase lock value for each subject's EEG signal and HAD signal across different EEG channels is as follows:
[0009] In the formula, The phase-locking values of the EEG signals and heading angle signals for different EEG channels were used for each subject. This represents the number of time points within the time window. The phase difference between the pure EEG signal and the pure heading angle signal HDA from different EEG channels in complex form at each time point. The phase difference between pure EEG signals and pure HDA signals from different EEG channels at each time point. For an instantaneous time point within the time window, It is the imaginary unit.
[0010] The formula for calculating the Hilbert transform is as follows:
[0011] In the formula, To analyze the signal, Indicates instantaneous phase, Indicates instantaneous amplitude. This indicates a pure EEG signal or a pure HDA signal; For time variables, The imaginary unit, This is the Hilbert transform operation.
[0012] The phase difference between the different EEG channels and the heading angle channel at each time point is calculated using the following formula:
[0013] In the formula, This represents the instantaneous phase of different brainwave channels at time t. This represents the instantaneous phase of channels with different heading angles at time t.
[0014] The significance analysis of the phase lock values of different EEG channels and the heading angle signal for each subject was specifically performed by: pairing samples of the phase lock values of different EEG channels and the heading angle signal for each subject in normal and fatigued states. The test is performed using the following formula:
[0015]
[0016]
[0017]
[0018] In the formula, The sample standard deviation represents the difference. The sample mean of the difference. This represents the difference between each pair of data. and Indicates the first For two observations in the data, This represents the logarithm of the paired samples; using the above formula, paired samples are generated for the phase-locked values of the EEG signals from different EEG channels and the heading angle signal for each subject in both normal and fatigued states. The phase-locking values of the current EEG signals and heading angle signals in the normal and fatigued states with significant differences were screened out. Finally, the phase-locking values of the current EEG signals and heading angle signals in the normal and fatigued states with significant differences among all subjects were determined as the target set, which is the set of phase-locking values of the target EEG channels of all subjects.
[0019] The phase-locked value set of the target EEG channel for all subjects was labeled according to different states. Specifically, the phase-locked value set of the target EEG channel for all subjects was labeled with a state label according to different states. State label label=1 indicates that the subject is in a normal state, and state label label=0 indicates that the subject is in a fatigued state.
[0020] A support vector machine regression model is used. The set of phase-lock values of the target EEG channels of all subjects and their corresponding state labels are used to form a training set, which is then used as the input to the support vector machine regression model. The support vector machine regression model is trained to establish the mapping between phase-lock values and state labels, thereby obtaining a fatigue detection regression model.
[0021] A multimodal fatigue driving detection system based on EEG-heading angle fusion includes a data acquisition module, a phase analysis module, a feature selection module, and a fatigue detection module. The data acquisition module is used to acquire electroencephalogram (EEG) signals from different subjects in normal and fatigue states. and heading angle signal ; The phase analysis module is used to analyze the electroencephalogram (EEG) signals of different subjects in normal and fatigue states. and heading angle signal The instantaneous phase is extracted by Hilbert transform, and the phase difference at each time point is calculated based on the instantaneous phase to obtain the phase lock value between the EEG signal of different EEG channels and the heading angle signal for each subject. The feature filtering module is used to perform a significance analysis on the phase lock values of the EEG signals and heading angle signals of different EEG channels for each subject, obtain the set of phase lock values of the target EEG channels for all subjects, and label the states according to different states. The fatigue detection module is used to construct a fatigue detection regression model, which detects fatigue in subjects based on the set of phase-locked values of the target EEG channels of all subjects and their corresponding different state labels.
[0022] Compared with the prior art, the present invention has the following beneficial effects: This invention proposes a multimodal fusion method for fatigue driving detection based on EEG and heading angle. It innovatively integrates EEG signals and heading angle signals to construct a dual-modal neurobehavioral detection system. Through phase synchronization analysis and optimized model architecture, it achieves accurate and efficient identification of fatigue states, providing a novel technical solution for driving safety. This invention has the following advantages: deep fusion of multimodal information; by calculating the phase-locking values of EEG signals from different EEG channels and heading angle signals, it achieves coupled analysis of physiological state and external behavior, significantly outperforming single-modal detection; innovative feature extraction method; based on Hilbert transform-based phase synchronization analysis, it extracts instantaneous phase through Hilbert transform, calculates the phase difference to obtain the phase-locking value, greatly reducing feature dimensions and improving computational efficiency and system real-time performance.
[0023] Furthermore, this invention collects the driver's brain signals using an EEG cap and acquires the heading angle signal using a heading angle acquisition device, achieving multimodal fusion detection of EEG and heading angle signals. This overcomes the limitations of existing technologies that rely excessively on a single internal physiological signal. It captures changes in the driver's internal neurophysiological activity and reflects functional deterioration in external driving behavior. These two signals complement each other from different dimensions to confirm fatigue state characteristics, enabling fatigue detection to cover both physiological and behavioral levels. This significantly improves the comprehensiveness and accuracy of the detection and solves the technical defect that a single modality cannot fully characterize the fatigue state.
[0024] Furthermore, this invention employs Hilbert transform to extract the instantaneous phase of bimodal signals, overcoming the limitation of traditional signal analysis that only focuses on amplitude changes. It can accurately capture the dynamic phase characteristics of two types of signals changing over time. By calculating the phase difference at different time points and converting it into a complex number, and averaging it within a time window to obtain the phase-locked value, a quantitative characterization of the coupling relationship between EEG signals and heading angle signals is achieved, providing an intuitive and comparable core indicator for subsequent feature selection. By further selecting phase-locked values with significant differences, a fatigue detection regression model between neural activity and driving behavior is constructed, filling the gap in existing technologies that lack quantitative modeling of the correlation between the two. This allows fatigue detection results to be traced back to the synchronous change patterns of EEG signals and heading angle signals, no longer relying on abstract black-box model outputs, significantly improving the interpretability of the fatigue detection regression model, and making the logical basis of the detection results clearer and more traceable.
[0025] Furthermore, this invention implements a preprocessing procedure for EEG signals, including electrode localization, bandpass filtering, independent component analysis, and rereference. Bandpass filtering and data interpolation are applied to the heading angle signal to effectively remove noise interference and ensure the purity of the input data. Simultaneously, paired sample testing is used to screen core features, eliminate redundant information, and reduce model computational complexity. Combined with a support vector machine regression model, targeted training is performed using labeled state tags to obtain a fatigue detection regression model. This model can accurately learn the differences between fatigue and normal states, significantly improving the generalization ability and robustness of the fatigue detection regression model. It solves the problems of existing technologies having easily interfered detection results and limited applicability.
[0026] Furthermore, the fatigue driving detection system constructed in this invention achieves full automation of the entire process from signal acquisition, processing, feature extraction to fatigue determination through the coordinated operation of the data acquisition module, phase analysis module, feature selection module, and fatigue detection module. Data acquisition is completed using a PC, a driving simulator, an EEG cap, and a heading angle acquisition device. The equipment is conveniently deployed, the operation process is standardized, and detection work can be carried out without complex professional skills. The functions of each module are clearly defined and seamlessly connected, enabling efficient processing of multi-source data and output of detection results. This provides a practical technical solution for fatigue detection in real-world driving scenarios, enhancing the practicality and promotional value of the technology. Attached Figure Description
[0027] Figure 1 A schematic diagram of a multimodal fatigue driving detection system that integrates EEG and heading angle. Figure 2 This is a schematic diagram of a multimodal fusion method for detecting fatigue driving based on EEG and heading angle. Figure 3 The original acquired EEG signals and heading angle signal Schematic diagram; Figure 4 Schematic diagram of preprocessed clean EEG signal and clean heading angle signal HDA; Figure 5 This is a time-domain waveform feature diagram of the pure EEG signal and pure heading angle signal of the Cz EEG channel under non-fatigue conditions in an embodiment of the present invention; Figure 6 This refers to the phase difference between the pure EEG signal and the pure heading angle signal of the Cz EEG channel at each time point under non-fatigue conditions in this embodiment of the invention. Figure 7 The time-domain waveform characteristics of the pure EEG signal and the pure heading angle signal HDA of the Cz EEG channel under fatigue state are shown. Figure 8 The phase difference between the pure EEG signal and the pure heading angle signal in the Cz EEG channel at each time point under fatigue conditions; Figure 9 This is a channel distribution diagram of the phase lock value (PLV) between EEG signals and heading angle signals in different EEG channels under non-fatigue conditions. Figure 10 This is a channel distribution diagram of the phase lock value (PLV) between EEG signals and heading angle signals in different EEG channels under fatigue conditions. Figure 11 The graph shows a significant difference in the phase lock value (PLV) between EEG signals and heading angle signals from different EEG channels under non-fatigue and fatigued conditions. Figure 12The mean distribution of phase lock value (PLV) of EEG signals and heading angle signals from different EEG channels in a non-fatigue state; Figure 13 The mean distribution of phase lock value (PLV) between EEG signals and heading angle signals from different EEG channels under fatigue conditions. Figure 14 A significant analysis of the differences in phase lock values between EEG signals from different EEG channels and heading angle signals for each subject; Figure 15 A comparison of the distribution of the heading standard deviation of the pure heading angle signal HDA between groups under fatigue and non-fatigue conditions; Figure 16 A comparison of the smoothness distribution between groups of the heading standard deviation of the pure heading angle signal HDA under fatigue and non-fatigue conditions; Figure 17 This diagram illustrates the performance comparison of fatigue detection using an SVM (Support Vector Machine) classifier for three feature input schemes: pure heading angle signal HDA only, pure EEG signal only, and a combination of pure EEG signal and pure heading angle signal HDA only. Detailed Implementation
[0028] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.
[0029] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0030] Example 1 This embodiment proposes a multimodal fusion method for fatigue driving detection based on EEG and heading angle, including the following steps: Obtaining EEG signals from different subjects in normal and fatigue states. and heading angle signal ; Electroencephalogram (EEG) signals of different subjects in normal and fatigue states and heading angle signal The instantaneous phase is extracted by Hilbert transform, and the phase difference at each time point is calculated based on the instantaneous phase to obtain the phase lock value between the EEG signal of different EEG channels and the heading angle signal for each subject. Significance analysis was performed on the phase lock values of EEG signals and heading angle signals of different EEG channels for each subject to obtain the set of phase lock values of the target EEG channels for all subjects, and state labels were marked according to different states. A fatigue detection regression model was constructed, which was used to detect fatigue in subjects based on the set of phase-locked values of the target EEG channels of all subjects and their corresponding different state labels.
[0031] Based on the steps of the above-mentioned EEG-heading angle multimodal fusion fatigue driving detection method, combined with Figure 2 The method flowchart shown in this embodiment is described in detail below, and the specific implementation method is as follows: This invention, based on the stability of the heading angle signal and the activity characteristics of specific frequency bands in EEG (such as the beta band), can effectively reveal the coupling relationship between behavior and neural activity. Studies have shown that the power of the EEG beta band is positively correlated with the stability of the heading angle signal; therefore, calculating the phase synchronization between the EEG signal and the heading angle signal has a clear theoretical basis. The method of this invention first synchronously acquires the EEG signal and the heading angle signal, performs a Hilbert transform on both signals, extracts their instantaneous phase, and constructs an analytical signal. Subsequently, the phase difference between the EEG signal and the heading angle signal in each EEG channel is calculated at time points and mapped to a complex form using an exponential function. Finally, within a set time window, the phase differences between the complex form of the EEG signal and the heading angle signal in each EEG channel are averaged to obtain a phase-locked value, thereby constructing a two-dimensional phase-locked value feature vector for quantifying the neuro-behavioral coupling strength.
[0032] Specifically, a simulated driving environment is constructed using a PC (Personal Compute Service) and a driving simulator platform. The PC controls the operation of the simulated driving scenario and synchronously receives and stores the collected signal data. The driving simulator platform provides operating logic consistent with real driving, ensuring that the subject's driving behavior closely matches the real-world scenario. A heading angle acquisition device is fixedly mounted on the driving simulator platform, and its signal output is connected to the PC's data interface. During the subject's driving, the heading angle signal is collected in real time when the vehicle turns, under both normal and fatigued conditions. To ensure synchronization of the two signals in time, each participant wore a professional EEG cap connected to a PC via a data transmission cable. During the participant's driving, EEG signals generated by neuronal activity in both normal and fatigued states were simultaneously collected. During the data acquisition process, it was ensured that the electrodes made good contact with the scalp to minimize signal interference. Through the above procedure, electroencephalogram (EEG) signals were collected from multiple subjects in both normal and fatigued states. like Figure 3 As shown in (a), the heading angle signal like Figure 3 As shown in (b), ensure that the sample size meets the needs of subsequent statistical analysis and model training.
[0033] Original EEG signals from different subjects in normal and fatigue states The signals are chaotic and disordered, with many poor channels, severely affecting signal quality and interfering with subsequent analysis processes; the heading angle signals of different subjects in normal and fatigued states... Sampling rate and EEG signals Inconsistencies necessitate the use of interpolation methods to unify the sampling rates, thereby aligning the two signals at different time points. Therefore, it is necessary to analyze the EEG signals of different subjects under normal and fatigued states. and heading angle signal Preprocessing is performed on the EEG signals; Electrode localization, bandpass filtering, independent component analysis, and rereference were performed on the preprocessed EEG signals. As the pure EEG signal ultimately used; for the heading angle signal Bandpass filtering and data interpolation are performed on the preprocessed heading angle signal. The final purified heading angle signal (HDA) is used. The preprocessed purified EEG signal is shown below. Figure 4 As shown in (a), the preprocessed clean heading angle signal HDA is as follows: Figure 4 As shown in (b), with Figure 3 Compared to the original signal shown, after the above-mentioned rereference and filtering preprocessing steps, the pure EEG signal is smoother and the quality is significantly improved; the features in the pure heading angle signal HDA are also clearer and more obvious.
[0034] Based on the preprocessed pure heading angle signal HDA, two derived behavioral features were extracted: heading standard deviation (HeadingStd) and smoothness. By comparing the inter-group distributions of fatigued and non-fatigue states, the state discrimination ability of the pure heading angle signal HDA was verified. This demonstrates that the preprocessed pure heading angle signal HDA can effectively characterize fatigue-induced driving behavior degradation, providing valuable behavioral data for subsequent phase synchronization analysis of pure EEG and pure heading angle signal HDA. Specifically, for example… Figure 15The figure shows a comparison of the distribution of the standard deviation of the heading angle signal HDA between fatigue and non-fatigue states. The standard deviation of the heading angle reflects the stability of steering wheel control by calculating the standard deviation of the heading angle sequence, which is the time-series data of the vehicle's forward direction angle relative to the geodetic coordinate system, synchronously collected by the onboard inertial measurement unit and GPS sensor. The results show that the median of the standard deviation of the heading angle is low and the data distribution is concentrated in the non-fatigue state, indicating that the driver's straight-line driving control is stable and the lateral fluctuation of the vehicle is small. In contrast, the median of the standard deviation of the heading angle increases significantly in the fatigue state, and the data dispersion increases, reflecting that fatigue leads to a decrease in the stability of the driver's steering wheel control. Figure 16 The figure shows a comparison of the smoothness distribution of the pure heading deviation signal HDA under fatigue and non-fatigue conditions. Smoothness reflects the smoothness of steering wheel operation and the precision of corrective actions. The median smoothness in the non-fatigue condition is high, and the data distribution is concentrated, indicating that the driver's directional control actions are smooth and the corrections are precise. Under fatigue conditions, the median smoothness is significantly lower, and there are more extreme values, revealing that fatigue impairs the smoothness of the driver's steering wheel operation, making directional corrections abrupt and coarse. This is a direct behavioral manifestation of fatigue-induced decreased motion control precision and weakened action planning ability.
[0035] The time-domain waveform characteristics of the pure EEG signal and the pure heading angle signal (HDA) of the Cz EEG channel were obtained under fatigue and non-fatigue states, respectively. The time-domain waveform characteristics of the pure EEG signal and the pure heading angle signal (HDA) of the Cz EEG channel under non-fatigue state are shown below. Figure 5 As shown, the pure EEG signal exhibits rich high-frequency components and large fluctuation amplitude, reflecting the brain's active cognitive processing and multi-task coordination ability in a conscious state; the pure yaw angle signal HDA maintains a stable and regular fluctuation pattern, reflecting precise driving control and stable behavioral output; the purity characteristics of both provide a reliable basis for subsequent phase extraction. The time-domain waveform characteristics of the pure EEG signal (EEG) and pure yaw angle signal (HDA) in the Cz EEG channel under fatigue state are shown in the figure below. Figure 7 As shown, the power of the theta band (4-8Hz) in the pure EEG signal is significantly enhanced, while the signal complexity is reduced, reflecting the neural mechanism of decreased brain activation and alertness levels under fatigue. Correspondingly, the pure heading angle signal HDA shows obvious increased volatility and pattern disorder, indicating impaired accuracy and stability of driving behavior.
[0036] Phase synchronization analysis was performed on the obtained pure EEG signal and pure heading angle signal (HDA). Hilbert transforms were then applied to both the EEG and HDA signals to obtain their respective analytic signals. Based on these analytic signals, the instantaneous phase of both signals was extracted. The Hilbert transform calculation formula is as follows:
[0037] In the formula, To analyze the signal, Indicates instantaneous phase, Indicates instantaneous amplitude. This indicates a pure EEG signal or a pure HDA signal; For time variables, The imaginary unit, This is the Hilbert transform operation.
[0038] Based on the instantaneous phase of the pure EEG signal and the pure HDA signal, the phase difference between different EEG channels and the HDA channel at each time point is calculated using the following formula:
[0039] In the formula, This represents the instantaneous phase of different brainwave channels at time t. This represents the instantaneous phase of channels with different heading angles at time t.
[0040] In this embodiment, the phase difference of the pure EEG signal and the pure heading angle signal HDA of the Cz EEG channel at each time point was collected under fatigue and non-fatigue conditions, and the distribution map of the phase difference was obtained; the phase difference distribution of the pure EEG signal and the pure heading angle signal HDA of the Cz EEG channel at each time point under the non-fatigue condition is as follows: Figure 6 As shown, in a non-fatigue state, the phase difference between the pure EEG signal and the pure yaw angle signal HDA in the Cz EEG channel at each time point corresponds to a point on the polar coordinate unit circle, exhibiting a relatively dispersed distribution overall. The phase lock value between the EEG signal and the yaw angle signal... The lower value indicates that, under conscious conditions, the neural activity of the motor cortex maintains a moderate degree of independence and flexibility from driving behavior, reflecting the nervous system's high adaptability and precise control ability in complex environments. The phase difference distribution of the pure EEG signal and pure yaw angle signal (HDA) of the Cz EEG channel at each time point under fatigue conditions is shown in the figure below. Figure 8As shown, under fatigue conditions, the phase difference distribution exhibits a significant concentration trend, with a marked increase in phase synchronization. This phenomenon indicates that as fatigue deepens, the coupling relationship between the cerebral cortex and behavioral output changes from flexible adaptation to rigid synchronization, reflecting a decline in the efficiency of neural resource integration and a weakening of cognitive control ability.
[0041] Furthermore, the phase difference between different EEG channels and the heading angle channel at each time point is converted into a complex number form. The modulus of the complex number is 1, and its direction is determined by the phase difference. The phase difference between different EEG channels and the heading angle channel in the transformed complex form falls within the range of each time point. Figure 6 On the unit circle shown, its direction is uniquely determined by the phase difference at the corresponding time point, thus completing the mathematical mapping of the phase difference from an angle value to a complex vector.
[0042] Then, within a time window, the phase difference between the complex number of different EEG channels and the heading angle channel at each time point is averaged to obtain the phase lock value of the EEG signal of different EEG channels and the heading angle signal for each subject; the calculation formula for the phase lock value of the EEG signal of different EEG channels and the heading angle signal for each subject is as follows:
[0043] In the formula, The phase-locking values of the EEG signals and heading angle signals for different EEG channels were used for each subject. This represents the number of time points within the time window. The phase difference between different EEG channels and the heading angle channel in complex form at each time point. The phase difference between different EEG channels and the heading angle channel at each time point. For an instantaneous time point within the time window, It is the imaginary unit.
[0044] Paired samples were generated for the phase-locked values of EEG signals from different EEG channels and the heading angle signal for each subject in both normal and fatigued states. The test is performed using the following formula:
[0045]
[0046]
[0047]
[0048] In the formula, The sample standard deviation represents the difference. The sample mean of the difference. This represents the difference between each pair of data. and Indicates the first For two observations in the data, This represents the logarithm of the paired samples; using the above formula, paired samples are generated for the phase-locked values of the EEG signals from different EEG channels and the heading angle signal for each subject in both normal and fatigued states. The test identifies phase-locked values of current EEG signals and heading angle signals in normal and fatigued states that show significant differences, ultimately determining the phase-locked values that show significant differences across all subjects. The target set is defined as the set of phase-locked values for the target EEG channels of all subjects. In this embodiment, =32, meaning 32 subjects; paired sample In the test, because the paired data has a mean constraint, it is necessary to use the degrees of freedom. After modification, the formula for calculating degrees of freedom is as follows: ,Right now Based on the above paired samples The test formula is obtained. The value, combined with the calculated value. Values and degrees of freedom Check The distribution table of values determines the corresponding probability value P; thus, the phase-locking value with significant differences is determined. That is, when the calculated result If the probability value P corresponding to the value is less than 0.05, then it is considered that there is a significant difference in the phase lock value between the current EEG signal and the heading angle signal in the normal state and the fatigue state.
[0049] Specifically, such as Figure 9 The figure shows the phase lock values between EEG signals from different EEG channels and the heading angle signal under non-fatigue conditions. Channel distribution map; Figure 10 The phase-locked values of EEG signals from different EEG channels and the heading angle signal under fatigue conditions. Channel distribution map; Figure 9 and Figure 10The horizontal axis in the diagram represents the standard EEG channel number of the international 10-20 EEG system. From left to right, the horizontal axes are as follows: Cz (Central z) corresponds to the central midline, located in the central region of the top of the head; Fz (Frontal z) corresponds to the frontal midline, located in the midline of the prefrontal lobe; Fp1 (Frontopolar 1) corresponds to the left frontal pole, the left anterior region of the prefrontal lobe; F7 (Frontal 7) corresponds to the lateral left frontal lobe, located in the lateral part of the left prefrontal lobe; F3 (Frontal 3) corresponds to the middle left frontal lobe, located in the middle region of the left prefrontal lobe; FC1 (Frontal-Central 1) corresponds to the left central frontal lobe, the junction of the left prefrontal lobe and the central region; C3 (Central 3) corresponds to the left central region, located in the central region of the left top of the head; FC5 ... central region of the left top of the head; Fp1 (Frontopolar 1) corresponds to the left central frontal lobe, the central region of the left frontal lobe, the central region of the left frontal lobe, the central region of the left frontal lobe, the central region of the left frontal lobe, the central region of the left frontal lobe, the central region of the left frontal lobe, the central region of the left frontal lobe, the central region of the left frontal lobe, the central region of the left frontal lobe, the central region of the left frontal lobe, the central region of the left frontal lobe, the central region of 5) Corresponding to the left frontocentral region (lateral), it is the lateral junction of the left prefrontal lobe and the central region; FT9 (Frontal-Temporal 9) corresponds to the left frontotemporal region (extremely lateral), located in the extremely lateral region of the left prefrontal lobe and temporal lobe; T7 (Temporal 7) corresponds to the left temporal lobe, located in the region near the left temple; CP5 (Central-Parietal 5) corresponds to the left central-parietal region, it is the junction of the left central region and parietal lobe; CP1 (Central-Parietal 1) corresponds to the left central-parietal region (medial), located in the medial junction of the left central region and parietal lobe; P3 (Parietal 3) corresponds to the left parietal lobe, located in the parietal lobe region at the back of the left top of the head; P7 (Parietal 7) corresponds to the left parietotemporal region, it is the junction of the left parietal lobe and temporal lobe; PO9 (Parietal-Occipital 9) corresponds to the left occipital-parietal region (extremely lateral), located in the extremely lateral region of the left parietal lobe and occipital lobe; O1 (Occipital 1) Corresponds to the left occipital lobe, located in the visual center region of the left posterior lobes; Pz (Parietal z) corresponds to the midline of the parietal lobe, located at the midline of the parietal lobe; Oz (Occipital z) corresponds to the midline of the occipital lobe, located in the midline region of the occipital lobe (visual center); O2 (Occipital 2) corresponds to the right occipital lobe, the visual center region of the right posterior lobes; PO10 (Parietal-Occipital 10) corresponds to the right occipitoparietal region (extremely lateral), located in the extremely lateral region of the right parietal and occipital lobes; P8 (Parietal 8) corresponds to the right parietotemporal region, the junction of the right parietal and temporal lobes; P4 (Parietal 4) corresponds to the right parietal lobe, located in the parietal lobe region of the right posterior top of the head; CP2 (Central-Parietal 2) corresponds to the right central parietal region (medial), located at the medial junction of the right central region and the parietal lobe; CP6 (Central-Parietal 2) corresponds to the right central parietal region (medial), located at the medial junction of the right central region and the parietal lobe; 6) Corresponding to the right central parietal region (outer side), located at the junction of the right central region and the outer side of the parietal lobe;T8 (Temporal 8) corresponds to the right temporal lobe, located in the area near the right temple; FT10 (Frontal-Temporal 10) corresponds to the right frontotemporal region (extremely lateral), which is the extremely lateral region between the right prefrontal lobe and the temporal lobe; FC6 (Frontal-Central 6) corresponds to the right frontal central region (lateral), located at the lateral junction of the right prefrontal lobe and the central region; C4 (Central 4) corresponds to the right central region, located in the central region of the right top of the head; FC2 (Frontal-Central 2) corresponds to the right frontal central region, which is the junction of the right prefrontal lobe and the central region; F4 (Frontal 4) corresponds to the middle part of the right frontal lobe, located in the middle region of the right prefrontal lobe; F8 (Frontal 8) corresponds to the lateral part of the right frontal lobe, located in the lateral part of the right prefrontal lobe; Fp2 (Frontopolar 2) corresponds to the right frontal pole, which is the right side region at the very front of the prefrontal lobe. Phase lock values of EEG signals and heading angle signals from different EEG channels in non-fatigue and fatigue states were obtained respectively. From the channels, extract the phase-locked values of the EEG signals from different EEG channels and the heading angle signal. As observed value and ,Depend on Figure 9 It can be seen that, under non-fatigue conditions, the phase lock values between EEG signals from different EEG channels and the heading angle signal... Significant spatial heterogeneity was observed in different EEG channels: observations in the prefrontal and central channels (Cz, Fz, Fp1, etc.) Relatively high, observed values in some channels (CP5, P7, O1, etc.) of the parietal-occipital lobe and temporal region. Significantly reduced; this distribution pattern reflects that in the waking state, the neural activity in the central-frontal region of the brain is more phase-synchronized with heading control behavior, while the parietal-occipital region is more involved in visuospatial information processing and has a relatively weaker direct coupling with heading behavior.
[0050] Depend on Figure 10 As shown, compared with the non-fatigue state, the phase lock values of EEG signals from different EEG channels and the heading angle signal in the fatigue state are... The distribution pattern of prefrontal channels (Fp2, F4, F8, etc.) has changed significantly: the observed values of... Significantly increased, with observations in the central and top channels (Cz, CP2, P4, etc.) showing a rise. Although there were fluctuations, the overall pattern showed a prefrontal cortex-dominated high synchronicity. This suggests that under fatigue, the prefrontal cortex may compensate for the fluctuations in phase synchronicity in the central and parietal regions by enhancing phase synchronization with directional behavior, thus maintaining driving control function. For the 32 subjects, observations were constructed for all EEG channels. and Paired sample groups, complete the pairing of samples The data foundation for testing is established.
[0051] Phase-locked values of EEG signals and heading angle signals from different EEG channels under non-fatigue and fatigued states were selected. Key channels with significant differences, such as Figure 11 As shown. In channels Cz and F8, the phase-locked values of EEG signals and heading angle signals in different EEG channels during non-fatigue states are... The median values were significantly higher than those in the fatigue state, reflecting that these channels were more synchronized with heading behavior when awake, and the coupling degree decreased after fatigue. However, in channels such as F7 and FC1, the phase-locking values between the EEG signals of different channels and the heading angle signal in the fatigue state were... The median value surpassed that of the conscious state, reflecting the directional remodeling of brain region functional coupling caused by fatigue.
[0052] Acquire phase-locked values of EEG signals from different EEG channels and heading angle signals. Mean distribution plots for the non-fatigue state and the fatigue state, respectively, as shown below. Figure 12 and Figure 13 As shown. Figure 12 Phase-locked values of EEG signals from different EEG channels and heading angle signals The mean distribution map in the non-fatigue state shows the phase lock values between EEG signals from different EEG channels and the heading angle signal. Significant spatial heterogeneity was observed in different EEG channels: phase lock values between EEG signals and heading angle signals in different EEG channels of the central region (Cz, Fz), prefrontal cortex (Fp1, F3), and right frontotemporal region (F4, F8, Fp2) were observed. The values are at a relatively high level; while the phase-locked values of EEG signals from different EEG channels in the parietal-temporal region (such as CP5, P7, O1, etc.) with the heading angle signal are relatively high. The values are significantly reduced. This distribution pattern reflects that in the waking state, the neural activity synchronicity is stronger in the central-frontal region and the right frontotemporal region of the brain, while the parietal-temporal region is more involved in visual spatial information processing and has relatively weaker neural synchronicity.
[0053] Figure 13 Phase-locked values of EEG signals from different EEG channels and heading angle signals In the mean distribution map of the fatigue state, compared with the non-fatigue state, the phase lock values of EEG signals from different EEG channels and the heading angle signal in the fatigue state are... The channel distribution pattern has changed significantly: the phase lock values of EEG signals from different channels in the right frontotemporal region (F4, F8, Fp2) and some central-parietal regions (such as P4, CP2, CP6, etc.) with the heading angle signal have changed significantly. The values were significantly increased; while the phase-locked values of EEG signals from different EEG channels in the central region (Cz, Fz) and the left frontal region (F3, F7) with the heading angle signal were significantly increased. The value decreased. This change reveals the resource reallocation mechanism of neural synchronization in brain regions under fatigue, reflecting the characteristics of blurred functional specialization of brain regions and enhanced bilateral symmetry of neural synchronization caused by fatigue.
[0054] By performing a significance analysis on the phase lock values of the EEG signals and heading angle signals of different EEG channels for each subject, the following results were obtained: Figure 14 The significant difference analysis plot shown allows us to filter out the phase-locked values of EEG signals and heading angle signals from different EEG channels under non-fatigue and fatigued states. Key channels showing significant differences. Box plots visually illustrate the inter-group differences: in channels such as Cz and P8, the phase lock values between EEG signals and heading angle signals in different EEG channels under non-fatigue conditions... The median values were significantly higher than those in the fatigue state, indicating that these channels exhibited stronger neural synchronization during wakefulness and weakened synchronization after fatigue. Furthermore, in channels such as FC1, CP2, and F8, the phase-locked values between the EEG signals and the heading angle signal in different EEG channels during fatigue were significantly higher. The median value surpassed that of the conscious state, reflecting the directional remodeling of neural synchronization in brain regions caused by fatigue, especially the enhanced neural synchronization in the right frontotemporal region during fatigue.
[0055] The phase-locked value set of the target EEG channel for all subjects was labeled according to different states. Specifically, the phase-locked value set of the target EEG channel for all subjects was labeled with a state label. State label label=1 indicates that the subject is in a normal state, and state label label=0 indicates that the subject is in a fatigued state.
[0056] A support vector machine (SVM) regression model was used. The phase-locked values of the target EEG channels of all subjects and their corresponding state labels were divided into a training set and a test set in an 8:2 ratio. The training set was used as input to the SVM regression model to train it, establishing a mapping between phase-locked values and state labels. After training, a fatigue detection regression model was obtained. The test set was then input into the fatigue detection regression model to verify its effectiveness. Figure 17As shown, using classification accuracy as the core indicator, the fatigue detection performance of three feature input schemes—HDA (High-Definition Aspect Ratio) using only pure heading angle signals, EEG (Electroencephalogram) signals using only pure EEG signals, and a combination of EEG and HDA signals—was compared under an SVM classifier. The results show that HDA alone has limited detection performance, and a single heading behavior feature is far from sufficient to distinguish fatigue states. EEG alone significantly outperforms HDA alone, demonstrating the unique advantage of EEG signals in capturing the neural mechanisms of fatigue. The fusion scheme of EEG and HDA signals achieves the best performance, reaching 0.833. This verifies the complementarity of EEG and heading behavior features; EEG can reflect changes in neural activity during fatigue, while heading behavior can reflect the decline in operational performance due to fatigue. The fusion of these two features enables more comprehensive and accurate fatigue driving detection, providing strong performance support for the design of multimodal fatigue detection systems.
[0057] Example 2 This embodiment proposes a multimodal fatigue driving detection system based on EEG and heading angle fusion, such as... Figure 1 As shown, it includes a data acquisition module, a phase analysis module, a feature selection module, and a fatigue detection module; The data acquisition module is used to acquire electroencephalogram (EEG) signals from different subjects in normal and fatigue states. and heading angle signal ; The phase analysis module is used to analyze the electroencephalogram (EEG) signals of different subjects in normal and fatigue states. and heading angle signal The instantaneous phase is extracted by Hilbert transform, and the phase difference at each time point is calculated based on the instantaneous phase to obtain the phase lock value between the EEG signal of different EEG channels and the heading angle signal for each subject. The feature filtering module is used to perform a significance analysis on the phase lock values of the EEG signals and heading angle signals of different EEG channels for each subject, obtain the set of phase lock values of the target EEG channels for all subjects, and label the states according to different states. The fatigue detection module is used to construct a fatigue detection regression model, which detects fatigue in subjects based on the set of phase-locked values of the target EEG channels of all subjects and their corresponding different state labels.
[0058] The modules in the EEG-heading angle multimodal fusion fatigue driving detection system do not operate independently. Instead, they are linked by a real-time multi-source fusion data stream, with each module seamlessly connected through standardized data interfaces to meet the real-time requirements of driving scenarios. Leveraging its efficient data analysis capabilities and rapid response mechanism, the fatigue driving detection system can dynamically track driver fatigue during driving, proactively mitigating operational risks such as inattention and slow reaction times caused by fatigue. This builds an active and reliable protective barrier for driving safety, thereby realizing a multimodal data fusion method based on the EEG-heading angle multimodal fusion fatigue driving detection system proposed in this invention, thus improving the performance of fatigue detection methods.
[0059] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that can be understood by those skilled in the art. The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.
Claims
1. A multimodal fusion method for fatigue driving detection based on EEG and heading angle, characterized in that, Includes the following steps: Obtaining EEG signals from different subjects in normal and fatigue states. and heading angle signal ; Electroencephalogram (EEG) signals of different subjects in normal and fatigue states and heading angle signal The instantaneous phase is extracted by Hilbert transform, and the phase difference at each time point is calculated based on the instantaneous phase to obtain the phase lock value between the EEG signal and the heading angle signal of different EEG channels for each subject. Significance analysis was performed on the phase lock values of EEG signals and heading angle signals of different EEG channels for each subject to obtain the set of phase lock values of the target EEG channels for all subjects, and state labels were marked according to different states. A fatigue detection regression model was constructed, which was used to detect fatigue in subjects based on the set of phase-locked values of the target EEG channels of all subjects and their corresponding different state labels.
2. The method for fatigue driving detection based on EEG-heading angle multimodal fusion according to claim 1, characterized in that, The method involves acquiring electroencephalogram (EEG) signals from different subjects in normal and fatigued states. and heading angle signal Specifically, this involves using a PC and a driving simulator to simulate the driving process. A heading angle acquisition device is mounted on the driving simulator to collect heading angle signals during the driving process. The data collection involved different subjects wearing EEG caps while driving to collect brainwave signals. The collection.
3. The EEG-heading angle multimodal fusion fatigue driving detection method according to claim 1, characterized in that, The electroencephalogram (EEG) signals of different subjects in normal and fatigue states were analyzed. and heading angle signal Before extracting the instantaneous phase using Hilbert transform, the EEG signals of different subjects in normal and fatigue states were analyzed. and heading angle signal Preprocessing is performed, specifically: processing the EEG signals Electrode localization, bandpass filtering, independent component analysis, and rereference were performed on the preprocessed EEG signals. As the pure EEG signal ultimately used; for the heading angle signal Bandpass filtering and data interpolation are performed on the preprocessed heading angle signal. The final pure heading angle signal (HDA) is used.
4. The EEG-heading angle multimodal fusion fatigue driving detection method according to claim 3, characterized in that, Hilbert transforms were performed on the pure EEG signal and the pure HAD signal to obtain their respective analytical signals. Based on the respective analytical signals, the instantaneous phases of the pure EEG signal and the pure HAD signal were extracted. Based on the instantaneous phase of the pure EEG signal and the pure heading angle signal (HDA), the phase difference between the pure EEG signal and the HDA signal at each time point is calculated for different EEG channels. The phase difference between the pure EEG signal and the HDA signal at each time point is then converted into a complex form. Within a time window, the average of these complex phase differences between the pure EEG signal and the HDA signal at each time point is taken to obtain the phase lock value for each subject's EEG signal and heading angle signal for different EEG channels. The formula for calculating the phase lock value for each subject's EEG signal and heading angle signal is as follows: In the formula, The phase-locking values of the EEG signals and heading angle signals for different EEG channels were used for each subject. This represents the number of time points within the time window. The phase difference between the pure EEG signal and the pure heading angle signal HDA from different EEG channels in complex form at each time point. The phase difference between pure EEG signals and pure HDA signals from different EEG channels at each time point. For an instantaneous time point within the time window, It is the imaginary unit.
5. The EEG-heading angle multimodal fusion fatigue driving detection method according to claim 4, characterized in that, The formula for calculating the Hilbert transform is as follows: In the formula, To analyze the signal, Indicates instantaneous phase, Indicates instantaneous amplitude. This represents pure EEG signals and pure HDA signals; The imaginary unit, This is the Hilbert transform operation.
6. The EEG-heading angle multimodal fusion fatigue driving detection method according to claim 4, characterized in that, The phase difference between the pure EEG signal and the pure heading angle signal HDA at each time point in different EEG channels is calculated using the following formula: In the formula, This represents the instantaneous phase of different brainwave channels at time t. This represents the instantaneous phase of channels with different heading angles at time t.
7. The EEG-heading angle multimodal fusion fatigue driving detection method according to claim 1, characterized in that, The significance analysis of the phase lock values of different EEG channels and the heading angle signal for each subject was specifically performed by: pairing samples of the phase lock values of different EEG channels and the heading angle signal for each subject in normal and fatigued states. The test is performed using the following formula: In the formula, The sample standard deviation represents the difference. The sample mean of the difference. This represents the difference between each pair of data. and Indicates the first For two observations in the data, This represents the logarithm of the paired samples; using the above formula, paired samples are generated for the phase-locked values of the EEG signals from different EEG channels and the heading angle signal for each subject in both normal and fatigued states. The phase-locking values of the current EEG signals and heading angle signals in the normal and fatigued states with significant differences were screened out. Finally, the phase-locking values of the current EEG signals and heading angle signals in the normal and fatigued states with significant differences among all subjects were determined as the target set, which is the set of phase-locking values of the target EEG channels of all subjects.
8. The EEG-heading angle multimodal fusion fatigue driving detection method according to claim 7, characterized in that, The phase-locked value set of the target EEG channel for all subjects was labeled according to different states. Specifically, the phase-locked value set of the target EEG channel for all subjects was labeled with a state label. State label label=1 indicates that the subject is in a normal state, and state label label=0 indicates that the subject is in a fatigued state.
9. The EEG-heading angle multimodal fusion fatigue driving detection method according to claim 8, characterized in that, The process of constructing the fatigue detection regression model includes: using a support vector machine regression model, forming a training set of phase-locked values of the target EEG channels of all subjects and their corresponding state labels, using this as input to the support vector machine regression model, training the support vector machine regression model, establishing the mapping between phase-locked values and state labels, and obtaining the fatigue detection regression model.
10. A brainwave-heading angle multimodal fusion fatigue driving detection system, based on the brainwave-heading angle multimodal fusion fatigue driving detection method according to any one of claims 1 to 9, characterized in that, It includes a data acquisition module, a phase analysis module, a feature selection module, and a fatigue detection module; The data acquisition module is used to acquire electroencephalogram (EEG) signals from different subjects in normal and fatigue states. and heading angle signal ; The phase analysis module is used to analyze the electroencephalogram (EEG) signals of different subjects in normal and fatigue states. and heading angle signal The instantaneous phase is extracted by Hilbert transform, and the phase difference at each time point is calculated based on the instantaneous phase to obtain the phase lock value between the EEG signal of different EEG channels and the heading angle signal for each subject. The feature filtering module is used to perform a significance analysis on the phase lock values of the EEG signals and heading angle signals of different EEG channels for each subject, obtain the set of phase lock values of the target EEG channels for all subjects, and label the states according to different states. The fatigue detection module is used to construct a fatigue detection regression model, which detects fatigue in subjects based on the set of phase-locked values of the target EEG channels of all subjects and their corresponding different state labels.