A multi-modal physiological signal intelligent fatigue detection system and method

Abstract

This invention discloses an intelligent fatigue detection system and method for multimodal physiological signals, belonging to the field of artificial intelligence and computer application technology. The invention acquires multimodal physiological signals through a signal acquisition module, and after preprocessing and feature extraction, inputs them into a GRU time-series feature extraction module, a neural network static classification module, and a SOC energy analysis module, respectively. It innovatively introduces a dynamic gating fusion module, which adaptively adjusts the fusion weights of the three modules based on real-time signal quality and energy state, ultimately outputting the fatigue level and triggering an alarm. This invention solves the technical problems of existing multimodal fusion methods, such as fixed weights, lack of interpretability, and poor robustness in complex environments, achieving high-precision and high-reliability fatigue state recognition, and is suitable for various application scenarios such as driving safety and industrial monitoring.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer application technology, specifically relating to a multimodal physiological signal intelligent fatigue detection system and method. Background Technology

[0002] With the increasing application of artificial intelligence technology in the field of security monitoring, automatic fatigue state identification based on physiological signals has become a research hotspot. However, existing multimodal recognition schemes suffer from the following technical bottlenecks:

[0003] First, existing fusion methods are mostly static weighting or simple feature concatenation, such as the publicly available CNN-LSTM-Attention model. While this improves accuracy, it doesn't consider dynamic changes in signal quality, leading to a sharp performance drop when the signal is disturbed. Second, traditional deep learning models lack interpretability. For example, existing techniques using GRU networks for temporal modeling of multimodal physiological signals can capture temporal dependencies, but the model's decision-making process is like a "black box," making it difficult to provide a physiological-level state explanation. Furthermore, a single model cannot comprehensively characterize fatigue states. Backpropagation (BP) neural networks excel at handling static features but ignore temporal information; SOC energy models can provide explanations from an energy perspective, but their classification accuracy is insufficient when used alone. Summary of the Invention

[0004] To address the aforementioned technical problems in existing technologies, this invention proposes a multimodal physiological signal intelligent fatigue detection system and method, which solves the technical problems existing in current multimodal fatigue detection methods, such as fixed fusion weights, lack of signal reliability in calculation, and insufficient interpretability of physiological mechanisms.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: an intelligent fatigue detection system based on dynamic reliability gating fusion, comprising:

[0006] The signal acquisition module is configured to acquire multimodal physiological signals;

[0007] The signal preprocessing module, connected to the signal acquisition module, is configured to filter, remove artifacts, segment, and normalize the acquired multimodal physiological signals, and output a standardized signal matrix.

[0008] The feature extraction module, connected to the signal preprocessing module, is configured to extract time-domain, frequency-domain, and nonlinear features from the preprocessed signal and concatenate them into an overall feature matrix.

[0009] The signal quality assessment module, connected to the signal preprocessing module, is configured to assess the quality of each modal signal and output a reliability score.

[0010] The energy trend analysis module, connected to the feature extraction module, is configured to perform energy trend analysis based on the overall feature matrix and output energy critical indicators.

[0011] The GRU temporal feature extraction module, connected to the feature extraction module, is configured to extract temporal features and output the first fatigue probability.

[0012] The BP neural network static classification module, connected to the feature extraction module, is configured to perform static feature classification and output a second fatigue probability.

[0013] The SOC model output module, connected to the energy trend analysis module, is configured to output the third fatigue probability based on the energy critical index.

[0014] The dynamic reliability gating fusion module, which is connected to the signal quality assessment module, energy analysis module, time series modeling module, static classification module and energy probability mapping module, is configured to dynamically adjust the weights of the first fatigue probability, the second fatigue probability and the third fatigue probability according to the reliability score and the energy critical index, and perform weighted fusion to output the fused fatigue probability.

[0015] The classification alarm module, connected to the dynamic gating fusion module, is configured to output fatigue level based on the fused fatigue probability and trigger an alarm.

[0016] Preferably, the acquired multimodal physiological signals include at least two of the following: electroencephalography (EEG), eye movement (EOG), heart rate variability (HRV), and electromyography (EMG).

[0017] Preferably, the signal quality assessment module obtains a reliability score by calculating the signal-to-noise ratio and signal variance. The calculation formula is:

[0018] ;

[0019] in, For noise variance; The effective signal variance; To prevent small constants with a denominator of zero.

[0020] Preferably, the energy trend analysis module calculates the energy criticality index based on the self-organized criticality (SOC) theory. The calculation formula is:

[0021] ;

[0022] in, The power spectrum variance represents the energy fluctuation. Average energy; Power spectral skewness reflects the asymmetry of energy distribution.

[0023] Preferably, the GRU temporal feature extraction module is used to update the gated state and output the first fatigue probability, and its gated state update formula is:

[0024] ;

[0025] ;

[0026] ;

[0027] in, This is the current input vector, used to provide the multimodal physiological signal characteristics at this moment; , , The weight matrix is ​​input to each gate control unit and is used to characterize the influence of input features on the update gate, reset gate, and candidate state. The Sigmoid activation function is used to compress the linear combination result into the (0–1) interval to represent the fatigue probability; , , This is the weight matrix for the hidden state of the previous time step to each gating unit, used to describe the impact of historical states on the gating mechanism; This is the hidden state from the previous time step, used to carry fatigue-related information from the historical time step and affect the current gating calculation; To update the gate, control memory retention; To reset the door and control the forgetting rate; This is the current hidden state, which includes historical fatigue characteristics;

[0028] Change the current hidden state The input is fed into the classification layer to obtain the first fatigue probability. :

[0029] ;

[0030] in, For activation functions; This represents the current hidden state, used to characterize fatigue-related features in the time series up to this moment; This is the weight matrix of the GRU output layer, used to map the hidden state to linear features of fatigue probability; This is the bias term for the GRU output layer, used to shift and adjust the output result; GRU stands for Gated Loop Unit.

[0031] The first fatigue probability reflects the fatigue trend over time.

[0032] Preferably, the BP neural network static classification module outputs a second fatigue probability. The calculation formula is:

[0033] ;

[0034] in, The input feature vector of the BP network is used to provide static features of multimodal physiological signals; This is the first layer weight matrix, used to map input features to hidden layer features; This is the second layer weight matrix, used to map the hidden layer features to the output space; This is a bias term used to adjust the linear output of the network, giving the model higher expressive power. This is an activation function used for nonlinear transformation of input features; The sigmoid output function is used to restrict the output to the (0–1) interval to form the fatigue probability; BP represents the backpropagation neural network;

[0035] Preferably, the SOC model output module maps the energy critical index to the third fatigue probability through a single-layer sensing network, and the calculation formula is as follows:

[0036] ;

[0037] Among them, among them, It is an energy critical index used to characterize the relationship between changes in signal energy state and fatigue level; This is the weight matrix of the SOC output layer, used to map the energy criticality index to the fatigue probability space; This is a bias term for the SOC output layer, used to adjust the baseline level of the SOC model output; The Sigmoid activation function is used to map the linear transformation result to a fatigue probability of (0–1); This is the weight matrix. For bias terms;

[0038] ;

[0039] in, Power spectral density is used to characterize the energy distribution of physiological signals at different frequency components; The power spectral variance is used to measure the degree of fluctuation in energy distribution. This is the power spectrum mean, used to reflect the overall energy level; Power spectral skewness is used to describe the degree of skewness and asymmetry in the energy distribution. SOC is a comprehensive energy criticality index used to reflect the correlation between signal energy state and fatigue trend. SOC stands for self-organizing criticality.

[0040] Preferably, the dynamic reliability gating fusion module calculates the weights of each model through a gating network and performs weighted fusion, with the specific formula as follows:

[0041] ;

[0042] in, Let be the fusion weight of the i-th model, used to represent the contribution ratio of that model to the final output; The reliability score for the i-th model is used to reflect the quality or credibility of the input signal of the model. is the energy critical index for the i-th model, used to characterize the fatigue correlation of signal energy changes in this model; This is the gating parameter for the reliability channel, used to adjust the strength of the influence of the reliability score on the fusion weight; These are the gating parameters for the energy channel, used to adjust the influence of the energy critical index on the fusion weight. This is a gated bias term used to shift and adjust the calculation results of the fusion weights; The Sigmoid activation function is used to compress gated linear combination values ​​into the (0–1) interval to form fusion weights.

[0043] The final fusion output is:

[0044] ;

[0045] in, The fatigue probability output by the i-th model represents the independent judgment result of each model on the current fatigue state. For model indexing, , used to identify the three models participating in the fusion; The final fatigue probability is used to combine the outputs of the three models to obtain more stable and robust fatigue detection results.

[0046] Preferably, the classification alarm module maps the fused fatigue probability to fatigue levels, including conscious, mild fatigue, moderate fatigue, and severe fatigue, and triggers an audible and visual alarm when the fatigue level reaches a preset threshold.

[0047] Furthermore, this invention also mentions a multimodal physiological signal intelligent fatigue detection method, which employs the multimodal physiological signal intelligent fatigue detection system described above, specifically including the following steps:

[0048] Step 1: Collect multimodal physiological signals, including at least two of the following: electroencephalography (EEG), eye movement (EOG), heart rate variability (HRV), and electromyography (EMG);

[0049] Step 2: Filter, remove artifacts, segment, and normalize the multimodal physiological signals to obtain a standardized signal matrix;

[0050] Step 3: Extract time-domain, frequency-domain, and nonlinear features from the standardized signal matrix and concatenate them into a global feature matrix;

[0051] Step 4: Evaluate the quality of each modal signal and obtain a reliability score;

[0052] Step 5: Perform energy trend analysis based on the overall feature matrix to obtain the energy critical index;

[0053] Step 6: Extract temporal features to obtain the first fatigue probability;

[0054] Step 7: Perform static feature classification to obtain the second fatigue probability;

[0055] Step 8: Based on the energy criticality index, obtain the third fatigue probability;

[0056] Step 9: Dynamically adjust the weights of the first fatigue probability, the second fatigue probability, and the third fatigue probability based on the reliability score and energy criticality index, and perform weighted fusion to obtain the fused fatigue probability;

[0057] Step 10: Output fatigue level based on fusion fatigue probability and trigger alarm.

[0058] The beneficial technical effects of this invention are as follows:

[0059] 1. An innovative dynamic reliability-gated fusion mechanism is proposed, which combines signal quality assessment with energy state analysis to achieve adaptive adjustment of multi-model weights and solves the problem of poor robustness of traditional fixed-weight fusion in complex environments.

[0060] 2. A three-dimensional analysis framework of time series, static and energy was constructed. The GRU model captures the laws of temporal evolution, the BP model identifies static feature patterns, and the SOC model provides an explanation of energy trends, forming a complementary and synergistic identification system.

[0061] 3. Significantly improves the engineering practicality of the system. Under harsh conditions such as signal interference and partial mode loss, the system can automatically adjust the fusion strategy to maintain the stability of recognition performance, making it particularly suitable for complex working conditions such as underground and driving operations.

[0062] 4. The interpretability of the model has been enhanced. By introducing the SOC energy analysis module, a physiological explanation is provided for fatigue state determination, making the artificial intelligence decision-making process more transparent and credible. Attached Figure Description

[0063] Figure 1 This is a system overall structure block diagram of the present invention;

[0064] Figure 2 This is a schematic diagram of the working principle of the dynamic reliability gating fusion module;

[0065] Figure 3 This is a schematic diagram of the structure for the collaborative fusion of the three models;

[0066] Figure 4 This is a schematic diagram of SOC energy analysis and output principle. Detailed Implementation

[0067] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0068] This invention proposes a multimodal fatigue detection system and method based on Dynamic Reliability Gating (DRG) to solve the technical problems existing in current multimodal fatigue detection methods, such as fixed fusion weights, lack of signal reliability in calculation, and insufficient interpretability of physiological mechanisms.

[0069] This system integrates the GRU time-series feature extraction model, the BP neural network classification model, and the SOC energy trend analysis model. The structure of the three models working together is as follows: Figure 3 As shown, this invention proposes a method and system for detecting fatigue status of underground coal mine workers based on GRU–BP–SOC dynamic reliability gating fusion. This scheme achieves real-time identification and audible and visual alarms for the fatigue status of workers, drivers, or industrial operators through multimodal physiological signal acquisition, energy criticality analysis, time series modeling, dynamic weight fusion, and depth discrimination.

[0070] This invention has the following features:

[0071] Introducing a signal quality sensing mechanism improves the system's noise immunity and stability;

[0072] Establish a physiological energy threshold index based on SOC theory to improve model interpretability;

[0073] Dynamic adaptive fusion of multimodal features is achieved by employing the DRG gating mechanism;

[0074] It outputs real-time fatigue level and alarm signals for easy monitoring and intervention.

[0075] like Figure 1 As shown, the system of the present invention mainly includes the following modules:

[0076] Data acquisition module: Acquires physiological signals such as EEG, EOG, HRV, and EMG;

[0077] Signal preprocessing module: filtering, artifact removal, segmentation, normalization;

[0078] Feature extraction module: Extracts time-frequency features of multimodal signals;

[0079] Signal quality assessment module: Calculates the signal reliability score R;

[0080] SOC Energy Trend Analysis Module: Calculates signal reliability score R;

[0081] GRU time series modeling module: extracts time series features and predicts time-dependent fatigue trends;

[0082] BP neural network module: static feature classification, output fatigue probability;

[0083] SOC model output module: Calculates fatigue trend probability based on energy index;

[0084] Dynamic Reliability Gated Fusion Module (DRG): Fuses the outputs of the GRU-BP-SOC three models, and outputs the fatigue level and visualizes the results through the adaptive weighted classification and alarm visualization module.

[0085] (1) Data acquisition module;

[0086] This system is used for real-time acquisition of multimodal physiological signals from downhole workers, including electroencephalography (EEG), eye movement (EOG), heart rate (HRV), and electromyography (EMG). The sensor, mounted on a portable acquisition end, is attached to a safety helmet, chest strap, or eyeglass holder. The sampling frequency is 128–512 Hz, and data is transmitted to the signal processing unit via Bluetooth or wired connection. The sampled data is denoted as:

[0087] ;

[0088] in, This represents the signal sequence of the i-th mode.

[0089] (2) Signal preprocessing module; The original signal often contains motion artifacts and environmental noise. To ensure feature stability, this module is used to denoise, segment, and normalize the signals of each mode. Specifically, it includes: bandpass filtering (0.5–45 Hz) to remove power frequency and DC drift; independent component analysis (ICA) to remove eye movement and electromyography artifacts; sliding window segmentation (window width 5 s, step size 1 s); Z-score normalization:

[0090] ;

[0091] Normalization can eliminate individual differences and improve the generalization ability of the model.

[0092] Output multimodal normalized signal matrix:

[0093] .

[0094] (3) Feature extraction module;

[0095] This module extracts feature vectors from each modal signal to characterize different physiological activity patterns, as shown in Table 1:

[0096] Table 1. Characteristics of multimodal physiological signals and their correlation with fatigue;

[0097] .

[0098] Multimodal features are concatenated into a global feature matrix:

[0099] ;

[0100] These features are simultaneously input into the GRU, BP, and SOC modules for analysis.

[0101] (4) Signal quality assessment module; Signals of different modes are subject to varying degrees of interference at different times. This module assesses signal quality using signal-to-noise ratio and variance characteristics to obtain a reliability score. .

[0102] ;

[0103] in, For noise variance; The effective signal variance; To prevent small constants with a denominator of zero.

[0104] when A higher value indicates good signal quality;

[0105] when A lower value indicates that the signal is significantly affected by noise.

[0106] During the fusion phase, the system automatically reduces the weight of the model corresponding to low-quality signals.

[0107] (5) SOC energy trend analysis module; such as Figure 4As shown. This module is based on the theory of self-organized criticality (SOC) and is used to describe the critical balance between energy accumulation and release in physiological systems. During the process of transitioning from wakefulness to fatigue, the energy distribution of brain electrical and heart rate signals gradually concentrates and fluctuates more. Therefore, the fatigue trend can be quantified using the statistical properties of power spectral density (PSD).

[0108] The energy critical index is defined as:

[0109] ;

[0110] in, The power spectrum variance represents the energy fluctuation. Average energy; Power spectral skewness reflects the asymmetry of energy distribution.

[0111] when A large value indicates energy accumulation and that the system is approaching its fatigue critical point. This result serves as input for subsequent SOC modeling and DRG module calculations.

[0112] (6) GRU (Gated Cyclic Unit) temporal feature extraction module;

[0113] The GRU module is primarily used to extract time-series features of the signal to capture the gradual evolution of fatigue states. At each time step t, the GRU model updates the gated state as follows:

[0114] ;

[0115] ;

[0116] ;

[0117] in, To update the gate, control memory retention; To reset the door and control the forgetting rate; This is the current hidden state, which includes historical fatigue characteristics.

[0118] Output hidden vector Inputting the data into the classification layer yields the first fatigue probability:

[0119] ;

[0120] in, For activation functions; This represents the current hidden state, used to characterize fatigue-related features in the time series up to this moment; This is the weight matrix of the GRU output layer, used to map the hidden state to linear features of fatigue probability; This is the bias term for the GRU output layer, used to shift and adjust the output result; GRU stands for Gated Loop Unit.

[0121] The first fatigue probability reflects the fatigue trend over time.

[0122] (7) BP Neural Network Classification Module; The BP module takes static features as input and performs fast classification.

[0123] Its structure is a multilayer perceptron (MLP):

[0124] ;

[0125] in, Network weights; For activation functions; This is the output layer for the Sigmoid function.

[0126] The BP module can identify static fatigue signal patterns, demonstrating significant effectiveness in short-term identification and sudden fatigue detection. Combined with the GRU, it can simultaneously consider both short-term changes and long-term trends.

[0127] (8) SOC model output module; such as Figure 4 As shown.

[0128] This module takes the energy analysis results as input, combines the SOC theory to calculate the system's energy critical state, and outputs the fatigue trend probability.

[0129] 1. Input signal energy characteristics: PSD sequences from EEG, HRV, and other modes. 2. Calculation of energy critical index:

[0130] ;

[0131] in, Power spectral density is used to characterize the energy distribution of physiological signals at different frequency components; The power spectral variance is used to measure the degree of fluctuation in energy distribution. This is the power spectrum mean, used to reflect the overall energy level; Power spectral skewness is used to describe the degree of skewness and asymmetry in the energy distribution. SOC is a comprehensive energy criticality index used to reflect the correlation between signal energy state and fatigue trend. SOC stands for self-organizing criticality.

[0132] 3. Energy probability mapping layer;

[0133] Mapped to probability values ​​through a single-layer perceptron network:

[0134] ;

[0135] in , representing the probability that the system is in a fatigue critical state.

[0136] when At this point, the system is considered to have entered a mild fatigue stage;

[0137] when At that time, it was close to the critical point of severe fatigue.

[0138] The results can be used alone for fatigue trend analysis, or as input for the energy dimension of the DRG module.

[0139] The advantages of this module are: it establishes a correspondence between physiological signals and critical energy states; it gives the model interpretability; and it makes up for the "black box" problem of neural networks.

[0140] (9) Dynamic reliability gating fusion module (DRG unit);

[0141] Module principle as follows Figure 2 As shown, the goal is to dynamically adjust the fusion weights of the three models when the signal state changes, thereby improving the system's robustness and accuracy. The fusion weights are determined by the signal reliability... With energy indicators Joint decision:

[0142] ;

[0143] in: Let be the fusion weights for model i; is the gating parameter; b is the bias term.

[0144] The final fusion output is:

[0145] ;

[0146] Gated networks use Sigmoid normalization to ensure that the weights of each model change dynamically but the sum does not exceed 1.

[0147] Logical explanation:

[0148] When the signal quality is high and the energy is stable, increase the GRU weights;

[0149] When static features are significant, increase the weight of BP; when energy fluctuates abnormally, increase the weight of SOC.

[0150] In this way, the system can maintain a stable output even in the event of signal fluctuations, loss, or noise.

[0151] (10) Classification and alarm visualization module;

[0152] The system will merge the output. The mapping to fatigue levels is shown in Table 2:

[0153] Table 2 Fatigue Level Table;

[0154] .

[0155] When the level is ≥2, a light alarm is triggered, and the following information is displayed on the terminal:

[0156] Modal weight change trend, signal quality score of each mode, SOC energy fluctuation curve, and real-time fatigue level time series.

[0157] Key technical points (core innovative modules) of this invention:

[0158] 1) Signal quality-driven weight adjustment mechanism;

[0159] This invention calculates the reliability score of each modality signal through a signal quality assessment module, which serves as a dynamic control parameter in the fusion process. When a signal is affected by noise, drift, or loss, the system can automatically reduce the decision weight of that modality to ensure that the overall recognition result is not affected by the failure of a single modality. This mechanism effectively improves the robustness and fault tolerance of the system in complex environments.

[0160] 2) Physiological driving mechanisms based on energy criticality indices;

[0161] The self-organized criticality (SOC) theory is introduced to describe the evolution of human fatigue from the perspective of energy accumulation and release. The system extracts the signal energy distribution characteristics through the energy trend analysis module, generates an energy criticality index, and converts it into a fatigue trend probability. This index serves as an important reference factor for the fusion weights, making the model's decision-making physiologically interpretable.

[0162] 3) Dynamic reliability gating fusion structure (DRG module);

[0163] The DRG module generates dynamic fusion weights based on signal reliability and energy state, enabling adaptive weighting of the outputs of GRU, BP, and SOC models. This gating structure continuously adjusts the weights of each model in real time during system operation, allowing the system to select the optimal model combination for discrimination based on the current signal state. This design overcomes the problem of fixed weighting coefficients in traditional multimodal fusion, achieving dynamism and intelligence in the fusion process.

[0164] 4) A synergistic, complementary, and integrated structure of the three models;

[0165] The GRU model is responsible for extracting time-dependent features and identifying the dynamic trends of fatigue states; the BP model is responsible for the nonlinear classification of static features, improving the detection response speed in a short time; and the SOC model characterizes the signal energy change pattern, providing a physiological explanation of fatigue trends. The three models work together through a unified gating structure, forming a complementary analysis mechanism in the three dimensions of time, features, and energy, which significantly improves the overall stability and detection accuracy of the system.

[0166] 5) Fatigue level classification and alarm mechanism;

[0167] The system automatically determines the fatigue level based on the fusion output results, classifying it into four levels: alert, mild fatigue, moderate fatigue, and severe fatigue.

[0168] When the fatigue level reaches a preset threshold (such as moderate or above), the system automatically triggers an audible and visual alarm and sends the information back to the monitoring terminal. This mechanism enables the system to have an immediate early warning function, ensuring safety and practicality.

[0169] 6) Enhanced interpretability and visual design;

[0170] During operation, the system can display the quality scores, energy states, and model weight changes of each modality signal in real time. Through the energy trend analysis results provided by the SOC module, users can understand the basis for model judgment and its physiological relevance, achieving system transparency from "outcome-oriented" to "causal traceability."

[0171] 7) Modular and scalable system architecture;

[0172] The system adopts a modular structure design, with each functional unit (such as signal acquisition, feature extraction, fusion module, etc.) being independent and having a unified interface. Modal sensors can be flexibly added or removed according to different application scenarios, or some models can be replaced (such as replacing the BP model with a lightweight network). The architecture supports embedded implementation, edge computing deployment, and centralized cloud monitoring, and has good engineering portability and expansion potential.

[0173] Advantages of this invention:

[0174] 1. The intelligence and adaptability of the fusion mechanism have been significantly improved;

[0175] Traditional multimodal methods often employ fixed weighting or static attention mechanisms, which cannot respond to changes in signal state.

[0176] This invention utilizes the DRG mechanism to dynamically optimize model weights by using signal quality and energy critical indicators as dual control factors. The system can automatically select the optimal mode combination, avoiding recognition bias caused by the failure of a single mode.

[0177] 2. Multiple models work together and complement each other, resulting in more comprehensive feature representation;

[0178] Existing methods typically rely solely on temporal models (such as LSTM or GRU) or static classification models (such as BP neural networks) for judgment, neglecting the feature complementarity between different models. This invention integrates three types of models: GRU, BP, and SOC.

[0179] (1) The GRU model captures time series features and reflects the dynamic evolution of fatigue state;

[0180] (2) The BP model identifies static statistical features, improving the response speed of short-term detection;

[0181] (3) The SOC model reveals the critical change law of physiological state from an energy perspective.

[0182] The three elements complement and integrate, enabling the system to simultaneously focus on information from three dimensions: time, statistics, and energy, significantly improving detection accuracy.

[0183] 3. The signal quality sensing mechanism improves robustness and reliability;

[0184] Traditional fusion algorithms are highly sensitive to signal noise, drift, and missing data, often leading to unstable classification results. This invention introduces a signal quality assessment module into the fusion process, dynamically suppressing noise-affected modes by calculating signal reliability scores. This design enables the system to maintain stable operation even in harsh environments such as downhole, high-temperature, and vibration conditions, demonstrating significantly better robustness than existing technologies.

[0185] 4. Physiological explainability and credibility are significantly enhanced;

[0186] Most deep learning methods lack interpretability, only providing probabilities of results without explaining the underlying reasons. This invention introduces the physiological energy criticality theory through the SOC energy model, establishing a quantitative link between the occurrence of fatigue and the energy accumulation phenomena of EEG and heart rate. The system can display energy trend indicators and changes in the weights of each modality on the interface, allowing users to intuitively understand the model's decision-making basis and enhancing the credibility of the results.

[0187] 5. Modular structure and strong scalability;

[0188] The system of this invention adopts a modular design, with each functional module (acquisition, feature extraction, model fusion, and decision output) having an independent interface. This allows for flexible expansion of sensor types or replacement of sub-models based on actual application scenarios. This feature enables the invention to be applied not only to driver fatigue detection but also to fields such as mine safety, aerospace, and motion monitoring.

[0189] 6. Excellent performance and application results;

[0190] Through comparative experiments, the fatigue recognition accuracy of the system of this invention is improved by approximately 15% to 20% in low signal-to-noise ratio environments, the response time is shortened by approximately 30% compared to traditional methods, and it also has a real-time early warning function. Its overall performance is superior to existing multimodal fusion methods, demonstrating high practicality and promising engineering applications.

[0191] This invention, through an innovative Dynamic Reliability Gated Fusion Mechanism (DRG), achieves structural synergy and complementarity between GRU, BP, and SOC models; functionally, it implements adaptive weighting driven by both signal quality and energy state; and effectively improves the accuracy, robustness, and interpretability of the detection system. This invention can be used not only as a standalone fatigue detection system but also as a core algorithm module embedded in other platforms within intelligent safety monitoring systems, possessing promising prospects for industrialization and widespread application in fields such as intelligent driving, safety monitoring, and health management.

[0192] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A multimodal physiological signal intelligent fatigue detection system, characterized in that, include: The signal acquisition module is configured to acquire multimodal physiological signals; The signal preprocessing module, connected to the signal acquisition module, is configured to filter, remove artifacts, segment, and normalize the acquired multimodal physiological signals, and output a standardized signal matrix. The feature extraction module, connected to the signal preprocessing module, is configured to extract time-domain, frequency-domain, and nonlinear features from the preprocessed signal and concatenate them into an overall feature matrix. The signal quality assessment module, connected to the signal preprocessing module, is configured to assess the quality of each modal signal and output a reliability score. The energy trend analysis module, connected to the feature extraction module, is configured to perform energy trend analysis based on the overall feature matrix and output energy critical indicators. The GRU temporal feature extraction module, connected to the feature extraction module, is configured to extract temporal features and output the first fatigue probability. The BP neural network static classification module, connected to the feature extraction module, is configured to perform static feature classification and output a second fatigue probability. The SOC model output module, connected to the energy trend analysis module, is configured to output the third fatigue probability based on the energy critical index. The dynamic reliability gating fusion module, which is connected to the signal quality assessment module, energy analysis module, time series modeling module, static classification module and energy probability mapping module, is configured to dynamically adjust the weights of the first fatigue probability, the second fatigue probability and the third fatigue probability according to the reliability score and the energy critical index, and perform weighted fusion to output the fused fatigue probability. The classification alarm module, connected to the dynamic gating fusion module, is configured to output fatigue level based on the fused fatigue probability and trigger an alarm. The GRU temporal feature extraction module is used to update the gated state and output the first fatigue probability. Its gated state update formula is as follows: ； ； ； in, This is the current input vector, used to provide the multimodal physiological signal characteristics at this moment; , , The weight matrix is ​​input to each gate control unit and is used to characterize the influence of input features on the update gate, reset gate, and candidate state. Use the Sigmoid activation function; , , This is the weight matrix for the hidden state of the previous time step to each gating unit, used to describe the impact of historical states on the gating mechanism; This is the hidden state from the previous time step, used to carry fatigue-related information from the historical time step and affect the current gating calculation; To update the gate, control memory retention; To reset the door and control the forgetting rate; This is the current hidden state, which includes historical fatigue characteristics; Change the current hidden state The input is fed into the classification layer to obtain the first fatigue probability. : ； in, This represents the current hidden state, used to characterize fatigue-related features in the time series up to this moment; is the weight matrix of the GRU output layer, used to map the hidden state to linear features of fatigue probability; This is the bias term for the GRU output layer, used to shift and adjust the output result; GRU stands for Gated Loop Unit. The first fatigue probability reflects the fatigue trend over time; The static classification module of the BP neural network outputs the second fatigue probability. The calculation formula is: ； in, The input feature vector of the BP network is used to provide static features of multimodal physiological signals; This is the first layer weight matrix, used to map input features to hidden layer features; This is the second layer weight matrix, used to map the hidden layer features to the output space; This is a bias term used to adjust the linear output of the network, giving the model higher expressive power. The sigmoid output function is used to restrict the output to the (0–1) interval to form the fatigue probability; BP represents the backpropagation neural network; The SOC model output module maps the energy criticality index to the third fatigue probability through a single-layer sensing network. The calculation formula is: ； in, It is an energy critical index used to characterize the relationship between signal energy state and fatigue level; This is the weight matrix of the SOC output layer, used to map the energy criticality index to the fatigue probability space; This is a bias term for the SOC output layer, used to adjust the baseline level of the SOC model output; SOC stands for Self-Organizing Criticality. ； in, Power spectral density is used to characterize the energy distribution of physiological signals at different frequency components; The power spectral variance is used to measure the degree of fluctuation in energy distribution. This is the power spectrum mean, used to reflect the overall energy level; Power spectral skewness is used to describe the degree of skewness and asymmetry in the energy distribution.

2. The intelligent fatigue detection system for multimodal physiological signals according to claim 1, characterized in that, The acquired multimodal physiological signals include at least two of the following: electroencephalography (EEG), eye movement (EOG), heart rate variability (HRV), and electromyography (EMG).

3. The intelligent fatigue detection system for multimodal physiological signals according to claim 1, characterized in that, The signal quality assessment module calculates the reliability score by measuring the signal-to-noise ratio and signal variance. The calculation formula is: ； in, For noise variance; The effective signal variance; To prevent small constants with a denominator of zero.

4. The intelligent fatigue detection system for multimodal physiological signals according to claim 1, characterized in that, The energy trend analysis module calculates energy criticality indices based on the self-organized criticality (SOC) theory. The calculation formula is: ； in, The power spectrum variance represents the energy fluctuation. Average energy; Power spectral skewness reflects the asymmetry of energy distribution.

5. The intelligent fatigue detection system for multimodal physiological signals according to claim 1, characterized in that, The dynamic reliability gating fusion module calculates the weights of each model through a gating network and performs weighted fusion. The specific formula is as follows: ； in, Let be the fusion weight of the i-th model, used to represent the contribution ratio of that model to the final output; The reliability score for the i-th model is used to reflect the quality or credibility of the input signal of the model. is the energy critical index for the i-th model, used to characterize the fatigue correlation of signal energy changes in this model; This is the gating parameter for the reliability channel, used to adjust the strength of the influence of the reliability score on the fusion weight; These are the gating parameters for the energy channel, used to adjust the influence of the energy critical index on the fusion weight. This is a gated bias term used to shift and adjust the calculation results of the fusion weights; The final fusion output is: ； in, The fatigue probability output by the i-th model represents the independent judgment result of each model on the current fatigue state. For model indexing, , used to identify the three models participating in the fusion; The final fatigue probability is used to combine the outputs of the three models to obtain more stable and robust fatigue detection results.

6. The intelligent fatigue detection system for multimodal physiological signals according to claim 1, characterized in that, The classification alarm module maps the fatigue probability to fatigue levels, including conscious, mild fatigue, moderate fatigue, and severe fatigue, and triggers an audible and visual alarm when the fatigue level reaches a preset threshold.

7. A multimodal physiological signal intelligent fatigue detection method, characterized in that, The multimodal physiological signal intelligent fatigue detection system according to any one of claims 1-6 specifically includes the following steps: Step 1: Collect multimodal physiological signals, including at least two of the following: electroencephalography (EEG), eye movement (EOG), heart rate variability (HRV), and electromyography (EMG); Step 2: Filter, remove artifacts, segment, and normalize the multimodal physiological signals to obtain a standardized signal matrix; Step 3: Extract time-domain, frequency-domain, and nonlinear features from the standardized signal matrix and concatenate them into a global feature matrix; Step 4: Evaluate the quality of each modal signal and obtain a reliability score; Step 5: Perform energy trend analysis based on the overall feature matrix to obtain the energy critical index; Step 6: Extract temporal features to obtain the first fatigue probability; Step 7: Perform static feature classification to obtain the second fatigue probability; Step 8: Based on the energy criticality index, obtain the third fatigue probability; Step 9: Dynamically adjust the weights of the first fatigue probability, the second fatigue probability, and the third fatigue probability based on the reliability score and energy criticality index, and perform weighted fusion to obtain the fused fatigue probability; Step 10: Output fatigue level based on fusion fatigue probability and trigger alarm.

Images