A dynamic monitoring system for brain fatigue state based on multimodal signal time series analysis

Abstract

This invention discloses a dynamic monitoring system for brain fatigue state based on multimodal signal temporal analysis, relating to the fields of biomedical engineering and intelligent monitoring technology. It includes a cloud management platform, which is communicatively connected to the following modules: a wearable acquisition terminal for real-time acquisition of multimodal EEG and near-infrared signals, and integrates temporal attention mechanisms and sliding window technology for adaptive signal enhancement and feature analysis. By introducing temporal attention mechanisms and sliding window real-time feature extraction technology, combined with temporal convolutional networks and hidden Markov models, this invention can perform temporal modeling of multimodal EEG and near-infrared signals, capturing the dynamic evolution of fatigue states in real time. This represents a technological leap from static segmented monitoring to continuous dynamic identification, significantly improving the timeliness and continuity of fatigue identification. It can provide timely warnings for sudden fatigue events, effectively overcoming the shortcomings of traditional methods such as monitoring lag and untimely warnings.

Description

Technical Field

[0001] This invention relates to the fields of biomedical engineering and intelligent monitoring technology, specifically to a dynamic monitoring system for brain fatigue state based on multimodal signal time-series analysis. Background Technology

[0002] With the accelerating pace of social life and increasing work pressure, mental fatigue has become a common phenomenon, significantly impacting an individual's work efficiency, learning ability, and quality of life. Scientific research shows that mental fatigue is not limited to simple physiological fatigue but also involves complex changes in psychological state. Therefore, monitoring mental fatigue can provide a basis for early intervention and optimization of the work environment.

[0003] For example, Chinese Patent Publication No. CN118319304A describes a method for monitoring brain fatigue by integrating electroencephalography (EEG) and functional near-infrared spectroscopy. Near-infrared spectroscopy can complement EEG technology, and by integrating the two technologies, the fatigue state of the brain can be more accurately identified.

[0004] Existing brain fatigue monitoring methods mostly employ static segmented feature extraction, which makes it difficult to achieve real-time dynamic continuous monitoring of fatigue states in complex task scenarios. This results in the inability to provide early warnings for sudden fatigue events and significant monitoring lag. Furthermore, multimodal signals are susceptible to motion artifacts and sudden noise interference in high-intensity dynamic environments, causing signal distortion in monitored individuals during rapid movement and a sharp decline in fatigue recognition accuracy. Moreover, the lack of personalized adaptive capabilities across users and multiple scenarios, as well as edge-cloud collaborative optimization mechanisms, makes it difficult to dynamically adjust the model based on user physiological differences and environmental changes. This leads to unstable recognition performance, weak system generalization ability, and difficulty in supporting reliable brain fatigue state monitoring in real and complex scenarios. Therefore, this paper proposes a dynamic brain fatigue state monitoring system based on multimodal signal time series analysis to address the aforementioned problems. Summary of the Invention

[0005] To solve the above technical problems, the present invention is implemented through the following technical solution: a dynamic monitoring system for brain fatigue state based on multimodal signal time series analysis, comprising a cloud management platform, wherein the cloud management platform is communicatively connected to the following modules: Wearable acquisition terminal is used to acquire multimodal EEG and near-infrared signals in real time, and integrates temporal attention mechanism and sliding window technology for adaptive signal enhancement and feature analysis to obtain stable and high-quality fatigue signal features in high dynamic environment, thereby achieving stable and high-quality signal acquisition and analysis in high dynamic environment. The edge computing module combines temporal convolutional networks and hidden Markov models to perform temporal modeling and real-time identification of fatigue states at the device edge, thereby improving the real-time performance and response speed of fatigue state identification. The terminal display and early warning module decodes the state and predicts the trend of fatigue early warning signals based on the Hidden Markov Model, provides real-time early warning and risk alerts for sudden fatigue events, and displays fatigue early warning signals, attention changes and historical trends in real time, realizing the visualization of fatigue state and real-time trend prediction and early warning. The interactive early warning feedback module is used to receive user feedback, including confirming fatigue status and adjusting early warning thresholds, collaboratively optimizing human-machine monitoring strategies, supporting user interactive feedback, optimizing monitoring strategies, and improving the system's adaptability. The edge-cloud collaborative optimization module, based on the federated learning framework and ant colony optimization algorithm, performs multi-user model aggregation and optimization in the cloud and personalized model tuning on the terminal, realizing dynamic model updates and collaborative optimization across scenarios and users, and enabling cross-user and cross-scenario model collaborative optimization and personalized dynamic updates.

[0006] Preferably, the wearable acquisition terminal includes an adaptive anti-interference sensing unit and a local dynamic feature extraction unit; The adaptive anti-interference sensing unit is used to combine deep residual network and wavelet adaptive noise reduction technology to perform feature analysis on the captured multimodal EEG and near-infrared signals, suppress motion artifacts and sudden noise in real time, improve the signal-to-noise ratio, effectively suppress motion artifacts and sudden noise interference, and significantly improve the signal-to-noise ratio. The local dynamic feature extraction unit is used to extract fatigue signal features from multimodal EEG and near-infrared signals by employing sliding window real-time feature extraction technology combined with temporal attention mechanism, thereby achieving real-time and adaptive extraction of signal features and enhancing temporal expression capabilities.

[0007] Preferably, the adaptive anti-interference sensing unit performs the following steps: Wavelet adaptive denoising technology was used to decompose the captured multimodal raw EEG and near-infrared signals at multiple scales, separating high-frequency noise components and low-frequency physiological signal components, achieving preliminary signal-noise separation. This effectively removed high-frequency interference from the original signal and preserved the main physiological signal components, laying the foundation for subsequent accurate identification. A noise feature recognition model is constructed based on a deep residual network. The high-frequency noise components are identified and classified to distinguish motion artifacts, power frequency interference and sudden impulse noise. This enables intelligent identification of noise types, provides an accurate basis for targeted filtering, and improves the system's anti-interference capability in complex environments. The filtering parameters are dynamically adjusted based on the noise classification results. An adaptive filter is used to reconstruct the signal, suppress the identified noise components, and output a multimodal physiological signal with improved signal-to-noise ratio. Through the dynamically adapted filtering strategy, the signal quality is significantly improved, ensuring the accuracy and reliability of subsequent fatigue feature extraction.

[0008] Preferably, the local dynamic feature extraction unit performs the following steps: A sliding window is used to segment the denoised multimodal signal. The window length and step size are dynamically adjusted according to the task status to achieve real-time streaming processing of the signal. This achieves adaptive time resolution and balances the stability of steady-state characteristics with the speed of transient response. Within each window, a temporal attention mechanism is used to extract weighted features of the power, approximate entropy, and sample entropy of the α, β, and θ bands of the EEG signal, as well as the mean, peak value, slope, and variance of the HbO of the near-infrared signal. This effectively enhances key features related to fatigue evolution and suppresses transient noise interference. The weighted features are concatenated and normalized to form a fatigue signal feature sequence, which serves as the input feature set for fatigue state analysis. This constructs a standardized feature vector that integrates multimodal information, providing high-quality input for subsequent time series modeling.

[0009] Preferably, the edge computing module includes a time-series dynamic modeling unit and a real-time early warning decision unit; The temporal dynamic modeling unit, based on temporal convolutional networks and hidden Markov models, performs temporal modeling on the fatigue signal features extracted from the wearable acquisition terminal, continuously identifies existing fatigue states, and realizes continuous and accurate temporal modeling and dynamic identification of fatigue states. The real-time early warning decision unit is used to determine the fatigue state in real time based on the time-series modeling results, generate a fatigue early warning signal in combination with early warning rules, and encapsulate it into a structured early warning message, thereby realizing real-time early warning decision-making and structured message encapsulation for fatigue state.

[0010] Preferably, the time-series dynamic modeling unit performs the following steps: Temporal convolutional networks are used to extract deep temporal dependencies from the input feature sequences, capturing the dynamic evolution patterns of fatigue states. Dilated convolutions effectively capture the long-term evolution of fatigue, improving the continuity of state recognition. The output features of the temporal convolutional network are input into the hidden Markov model to establish the state transition probability matrix and observation probability model between fatigue states, and output the observation sequence. The fatigue states include awake, mild fatigue and severe fatigue. Based on probability modeling, the logical rationality of fatigue state transitions is enhanced and instantaneous misjudgment is reduced. The observation sequence is decoded based on the Viterbi algorithm to output the current fatigue state sequence, and the fatigue state is continuously identified. Global optimal path decoding is used to ensure the temporal consistency of the state sequence and improve the reliability of the identification results.

[0011] Preferably, the real-time early warning decision unit performs the following steps: Based on the output fatigue state sequence, combined with the preset fatigue threshold and duration rules, it is determined whether the warning conditions are triggered, so as to realize the real-time and automatic judgment of fatigue state and effectively capture potential risk nodes. If multiple consecutive time slices are detected to be in a state of severe fatigue, or if the fatigue level rises sharply in a short period of time, an emergency fatigue warning signal is generated, enabling accurate identification and graded warning of two types of fatigue modes: continuous accumulation and rapid deterioration. The warning signal, its confidence level, timestamp, and fatigue trend information are encapsulated into a structured warning message and sent to the subsequent module, namely the terminal display warning module, forming a standardized and traceable warning information flow to support rapid intervention and post-event analysis.

[0012] Preferably, the terminal display warning module performs the following steps: After receiving structured early warning messages, the system uses a hidden Markov model to perform state backtracking and trend prediction, and plots fatigue change curves and state transition diagrams, achieving an intelligent leap from passive alarm to proactive prediction, thereby improving the predictability and scientific nature of fatigue management. The terminal interface displays the current fatigue status, warning level, attention concentration index and historical fatigue trend chart in real time, providing an intuitive and comprehensive information presentation to help operators clearly understand their own status changes and the basis for system judgment. When a sudden fatigue warning occurs, an audible and visual alarm is activated and a risk alert is pushed to prompt the user to take intervention measures. Through strong multi-sensory prompts, the high-risk state is ensured to be perceived in a timely and effective manner, prompting a prompt response.

[0013] Preferably, the interactive early warning feedback module performs the following steps: It provides a user interface for users to confirm or deny the fatigue status currently determined by the system, and supports manual adjustment of the warning sensitivity parameters, enabling rapid verification and real-time optimization of the system's judgment, significantly improving the human-computer interaction efficiency and user satisfaction of the monitoring system. Collect user feedback data and current environmental information to build a feedback dataset for model optimization and early warning rule tuning. Establish a high-quality labeled dataset with scene context to provide reliable data support for continuous and accurate optimization of models and rules. By dynamically adjusting the warning threshold and status judgment rules based on feedback data, the fatigue monitoring strategy is optimized through human-machine collaboration. This enables the system to adaptively learn user habits and environmental changes, continuously improving monitoring accuracy and the timeliness of warnings.

[0014] Preferably, the edge-cloud collaborative optimization module performs the following steps: In the cloud, a federated learning framework is used to securely aggregate local models from multiple users to generate a global fatigue recognition model, protecting user data privacy. By training and updating the model locally on multiple distributed terminals, collective knowledge is gathered to improve the performance of the global model, while ensuring that users' original physiological data is not uploaded or leaked. Ant colony optimization algorithm is used to fine-tune key parameters in the global fatigue recognition model, improving the model's generalization ability and recognition accuracy in different scenarios. This effectively avoids the tediousness and subjectivity of traditional manual parameter tuning, enabling the model to adapt to different working environments and individual differences, achieving better overall recognition results. The optimized global fatigue recognition model is then distributed to various terminals, and personalized fine-tuning is performed based on user local data and feedback. This enables a dynamic and adaptive cross-user and cross-scenario model update mechanism, generating personalized versions that fit individual user physiological characteristics and usage habits, achieving precise monitoring tailored to each user.

[0015] This invention provides a dynamic monitoring system for brain fatigue state based on multimodal signal time-series analysis. It has the following beneficial effects: (I) This dynamic monitoring system for brain fatigue state based on multimodal signal temporal analysis introduces temporal attention mechanism and sliding window real-time feature extraction technology, combined with temporal convolutional network and hidden Markov model, to perform temporal modeling of multimodal EEG and near-infrared signals, capture the dynamic evolution process of fatigue state in real time, realize the technical leap from static segmented monitoring to continuous dynamic recognition, significantly improve the timeliness and continuity of fatigue recognition, and provide timely early warning for sudden fatigue events, effectively overcoming the shortcomings of traditional methods such as monitoring lag and untimely early warning.

[0016] (II) The brain fatigue state dynamic monitoring system based on multimodal signal time series analysis adopts an adaptive anti-interference fusion algorithm that combines deep residual network and wavelet adaptive noise reduction technology. It can intelligently identify and suppress multiple interference sources such as motion artifacts, power frequency interference and sudden impulse noise, ensuring that stable and high-quality physiological signals can be obtained in highly dynamic and complex task scenarios such as rapid user movement, and greatly improving the signal-to-noise ratio and the robustness of the recognition model.

[0017] (III) This dynamic monitoring system for brain fatigue state based on multimodal signal time sequence analysis can receive direct confirmation or denial of fatigue judgment from users through interactive early warning feedback, and allow users to adjust the early warning sensitivity according to their own feelings. It incorporates the user's subjective experience into the optimization closed loop, so that the system can continuously adjust the early warning rules and model parameters based on real feedback, realizing the transformation of the monitoring strategy from one-size-fits-all to personalized and adaptive, and improving the user experience and system credibility. Attached Figure Description

[0018] Figure 1This is a schematic diagram of the workflow of a dynamic monitoring system for brain fatigue state based on multimodal signal time-series analysis according to the present invention. Figure 2 This is a data flow diagram of a dynamic monitoring system for brain fatigue state based on multimodal signal timing analysis according to the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1, please refer to Figure 1 , Figure 2 This invention provides a technical solution: a dynamic monitoring system for brain fatigue state based on multimodal signal time-series analysis, including a cloud management platform, which has the following modules for communication connection: The wearable acquisition terminal is used to acquire multimodal EEG and near-infrared signals in real time, and integrates temporal attention mechanism and sliding window technology for adaptive signal enhancement and feature analysis to obtain stable and high-quality fatigue signal features in high dynamic environment. It realizes stable and high-quality signal acquisition and analysis in high dynamic environment. The wearable acquisition terminal includes an adaptive anti-interference sensing unit and a local dynamic feature extraction unit. The adaptive anti-interference sensing unit combines deep residual networks and wavelet adaptive denoising technology to perform feature analysis on captured multimodal EEG and near-infrared signals. It suppresses motion artifacts and sudden noise in real time, improving the signal-to-noise ratio (SNR). The wavelet adaptive denoising technology performs multi-scale decomposition on the captured raw multimodal EEG and near-infrared signals, separating high-frequency noise components from low-frequency physiological signal components. This achieves preliminary signal-noise separation, effectively removing high-frequency interference from the original signal while retaining the principal components of the physiological signal, thus facilitating subsequent accurate identification. Laying the foundation, a noise feature recognition model is constructed based on a deep residual network to identify and classify the separated high-frequency noise components, distinguishing motion artifacts, power frequency interference, and sudden impulse noise, thereby achieving intelligent noise type discrimination. This provides an accurate basis for targeted filtering, improves the system's anti-interference capability in complex environments, dynamically adjusts filtering parameters based on noise classification results, and uses an adaptive filter to reconstruct the signal, suppressing the identified noise components and outputting a multimodal physiological signal with improved signal-to-noise ratio. Through a dynamically adapted filtering strategy, the signal quality is significantly improved, ensuring the accuracy and reliability of subsequent fatigue feature extraction. It should be noted that in actual operation, wavelet adaptive noise reduction processing was performed on the acquired multimodal raw EEG and near-infrared signals. The Daubechies4 (db4) wavelet basis function was selected, and the signal was decomposed into five levels of wavelet decomposition, decomposing the raw signal into approximation coefficients (low-frequency part) and detail coefficients (high-frequency part). The sampling frequency of the EEG signal was set to 256Hz, and the sampling frequency of the near-infrared signal was 10Hz. After wavelet transform, the first and second levels of detail coefficients mainly contain high-frequency noise components (frequency range >64Hz), while the third to fifth levels... The detail coefficients and approximation coefficients cover low-frequency physiological signal components (frequency range 0.5-30Hz), achieving preliminary separation of signal and noise. The separated high-frequency noise components are input into a deep residual network model for feature recognition and classification. This network adopts the ResNet-18 architecture, with the input layer receiving a 256×256 time-frequency map matrix consisting of 5 residual blocks. Each residual block contains two 3×3 convolutional layers with channel numbers of 64, 128, 256, and 512 respectively. The network output layer is a Softmax classifier, corresponding to three noise categories. The model was trained on three noise types: motion artifacts, power frequency interference (50Hz and its harmonics), and burst impulse noise. During training, a cross-entropy loss function and the Adam optimizer were used, with a learning rate of 0.001 and a batch size of 32. After 100 training epochs, the model achieved a classification accuracy of 94.2% for these three noise types on the validation set, providing accurate noise type discrimination for adaptive filtering. Based on the noise classification results from the deep residual network, the filtering parameters were dynamically adjusted to reconstruct the signal. For motion artifacts, an adaptive least mean square (LMS) filter was used. The step size parameter μ was set to 0.01, and the reference signal was provided by triaxial accelerometer data. For power frequency interference, a 50Hz notch filter was enabled, with a quality factor Q value set to 30 and a bandwidth of 4Hz. For sudden impulse noise, a medium filter was applied with a window length of 5 sampling points. The filtered sub-band signals were reconstructed by inverse wavelet transform. The same db4 wavelet basis function and 5-layer reconstruction algorithm as the decomposition stage were used in the reconstruction process. The final output was a multimodal physiological signal that was evaluated and displayed, effectively preserving the physiological characteristics of key frequency bands such as α, β, and θ. The local dynamic feature extraction unit employs a sliding window real-time feature extraction technique combined with a temporal attention mechanism to extract fatigue signal features from multimodal EEG and near-infrared signals. This achieves real-time, adaptive feature extraction, enhancing temporal expressiveness. The sliding window segments the denoised multimodal signal, with the window length and step size dynamically adjusted according to the task status, enabling real-time streaming processing of the signal. This achieves adaptive temporal resolution, balancing steady-state feature stability with transient response speed. Within each window, the temporal attention mechanism is used to extract weighted features from the α, β, and θ band power, approximate entropy, and sample entropy of the EEG signal, as well as the mean, peak, slope, and variance of the HbO of the near-infrared signal. This effectively enhances key features related to fatigue evolution and suppresses transient noise interference. The weighted features are then concatenated and normalized to form a fatigue signal feature sequence, serving as the input feature set for fatigue state analysis. This constructs a standardized feature vector that integrates multimodal information, providing high-quality input for subsequent temporal modeling. It should be noted that a sliding window technique is used to dynamically segment continuous EEG and near-infrared signals to achieve real-time streaming processing of the denoised multimodal signals. The window length and sliding step size are adaptively adjusted according to the current task load and the operator's real-time performance. During the normal steady-state task phase, the window length is set to 30 seconds and the step size is 5 seconds to ensure the smoothness of feature extraction. When the system detects an increase in task difficulty, a significant increase in reaction time, or increased operator activity indicated by external sensors through the built-in algorithm, the window length will be dynamically shortened to 10 seconds and the step size to 2 seconds to improve the system's response to rapidly changing fatigue states. Speed ​​is controlled by a high-precision system clock to synchronize the opening and closing of windows and ensure precise matching of EEG signals (256Hz) and near-infrared signals (10Hz) at the sampling point level during segmentation. Within each sliding window, multi-dimensional feature calculations are performed in parallel on the segmented signals. For EEG signals, the Welch periodogram method is used to calculate their power spectral density, specifically extracting the relative power values ​​of the α band (8-13Hz), β band (14-30Hz), and θ band (4-7Hz), and calculating the (θ+α) / β power ratio. Simultaneously, fixed parameters (embedding dimension m=2, tolerance r=0.2 times the standard) are used. The approximate entropy of the signal and the sample entropy are calculated. For near-infrared signals, based on the time series of oxyhemoglobin concentration changes, the mean, peak value, first-order linear fitting slope, and variance are calculated. Subsequently, a temporal attention mechanism is introduced to weight the extracted original features. This temporal attention mechanism is implemented by a lightweight one-dimensional convolutional network, whose input is the feature sequences of the current window and the two preceding windows. By calculating the autocorrelation of features in the time dimension, appropriate attention weights are assigned to each feature to enhance important features that are strongly correlated with the evolution of fatigue state and suppress interference caused by instantaneous fluctuations. The EEG features weighted by temporal attention are then processed. The near-infrared and near-infrared features are spliced ​​across modes to form a unified composite feature vector. This vector has a fixed dimension of 10 and includes, in sequence: weighted relative power of α, β, and θ, (θ+α) / β value, approximate entropy, sample entropy, mean HbO, peak HbO, slope HbO, and variance HbO. Subsequently, the feature vector is processed using the min-max normalization method. The upper and lower bounds of the normalization are derived from the current operator's personal baseline data, which is calculated from 5 minutes of resting data before the experiment. Finally, the normalized feature vectors output in chronological order constitute the fatigue signal feature sequence for fatigue state analysis. The edge computing module is used to combine temporal convolutional networks and hidden Markov models to perform temporal modeling and real-time identification of fatigue states at the device edge, reduce data transmission latency, and improve the real-time performance and response speed of fatigue state identification. The edge computing module includes a temporal dynamic modeling unit and a real-time early warning decision unit. The temporal dynamic modeling unit, based on temporal convolutional networks and hidden Markov models, performs temporal modeling on fatigue signal features extracted from wearable acquisition terminals, continuously identifying existing fatigue states. This achieves continuous and accurate temporal modeling and dynamic identification of fatigue states. The temporal convolutional network extracts deep temporal dependencies from the input feature sequence, capturing the dynamic evolution pattern of fatigue states. Dilated convolution effectively captures the long-term evolution of fatigue, improving the continuity of state identification. The output features of the temporal convolutional network are input into the hidden Markov model to establish a state transition probability matrix and observation probability model between fatigue states, outputting an observation sequence. Fatigue states include conscious, mild fatigue, and severe fatigue. Probabilistic modeling enhances the logical rationality of fatigue state transitions, reducing instantaneous misjudgments. The Viterbi algorithm decodes the observation sequence, outputting the current fatigue state sequence for continuous fatigue state identification. Global optimal path decoding ensures the temporal consistency of the state sequence, improving the reliability of the identification results. It should be noted that after obtaining the normalized 10-dimensional fatigue signal feature sequence, a temporal convolutional network (TCN) is used to extract deep temporal dependencies from the fatigue signal feature sequence. This network adopts a dilated causal convolutional structure to ensure that the processing follows the temporal order and that there is no information leakage. Specifically, the network contains three residual blocks, each consisting of two dilated causal convolutional layers. The dilation coefficient increases exponentially with the network depth (1, 2, 4 in sequence). The kernel size is uniformly set to 3, and the number of channels is fixed at 64. After each convolution, the ReLU activation function and layer normalization are applied, and the dropout rate is 0.A random dropout strategy of 1 is used to prevent overfitting. The network input is a sequence of length L (dynamically changing according to the sliding window strategy), with each time step corresponding to a 10-dimensional feature vector. TCN effectively expands the receptive field through its dilated convolution mechanism, abstracting and fusing contextual information from multiple time scales layer by layer, and outputting a 64-dimensional high-level feature sequence of the same length but containing deep temporal patterns. The 64-dimensional high-level feature sequence output by TCN is used as the observation sequence and input into a Hidden Markov Model (HMM) for probabilistic modeling. The model contains three hidden states, corresponding to wakefulness, mild fatigue, and severe fatigue, respectively. First, the model is trained using the Baum-Welch algorithm based on a large amount of labeled data (containing multimodal signal feature sequences of different fatigue stages). The model parameters are iteratively optimized. After training, the model learns a 3x3 state transition probability matrix A, which describes the probability of transitioning between the three fatigue states (but the probability of directly transitioning from wakefulness to severe fatigue is constrained to be extremely low), and an observation probability matrix B. This matrix defines the probability density of the current 64-dimensional observation features (fitted by a Gaussian mixture model) in each hidden state. In each step of the practical application stage, the model will be updated in real time. The obtained high-level feature vectors of the TCN are input into the trained HMM. A forward algorithm is used to calculate the probability of these vectors as observations of each hidden state, thus outputting a probability-based sequence of observed states corresponding to the current time step. This sequence reflects the internal fatigue state change process most likely to produce the given observation data. To determine the most probable true fatigue state sequence from the probabilities of the HMM output, a Viterbi algorithm is used for globally optimal path decoding. This algorithm, based on dynamic programming, traverses all possible state sequence paths, recursively calculating and retaining the maximum cumulative probability of reaching each time step and each state (this probability is determined by the state transition probability A, the observation probability B, and the initial state distribution π). Finally, it backtracks to obtain the state sequence with the highest cumulative probability over the entire time range. The decoding process follows the dynamic constraints of the trained HMM to ensure the temporal rationality of the output state sequence. The decoded state sequence is the continuous-time fatigue state evolution process ultimately determined by the system. This sequence recognition result serves as the core judgment basis and is transmitted in real-time to the real-time early warning decision unit to trigger different levels of early warning signals, achieving dynamic, continuous, and high-precision monitoring and early warning of the operator's mental fatigue state. The real-time early warning decision unit is used to judge the fatigue state in real time based on the time series modeling results. It generates fatigue early warning signals and encapsulates them into structured early warning messages in combination with early warning rules. This enables real-time early warning decision-making and structured message encapsulation for fatigue states. Based on the output fatigue state sequence and combined with preset fatigue degree thresholds and duration rules, it determines whether the early warning conditions are triggered, realizing real-time and automatic judgment of fatigue states and effectively capturing potential risk nodes. If multiple consecutive time slices are detected to be in a state of severe fatigue, or if the fatigue degree rises sharply in a short period of time, a sudden fatigue early warning signal is generated. This enables accurate identification and graded early warning for two types of fatigue modes: continuous accumulation and rapid deterioration. The early warning signal, its confidence level, timestamp, and fatigue trend information are encapsulated into structured early warning messages and sent to the subsequent module, namely the terminal display early warning module, forming a standardized and traceable early warning information flow to support rapid intervention and post-event analysis. It should be noted that, based on the real-time parsed fatigue state sequence, a built-in early warning decision engine is used to perform multi-level early warning judgments. Specifically, the early warning decision engine has a built-in dual trigger mechanism: continuous early warning and abrupt early warning. For continuous early warning, when the system detects that the operator's continuous time segments in a state of severe fatigue exceed a preset threshold (three consecutive time slices, i.e., a maximum of 6 seconds), a higher-level early warning is immediately triggered. This threshold is set based on empirical data from ergonomic research on the correlation between continuous cognitive decline and accident risk, and can be fine-tuned within a recommended range (two to four consecutive time slices) through the management interface. The decision-making process simultaneously calculates the confidence level of the state, which is derived from the probability value of the Hidden Markov Model decoding path and the matching degree between the current feature vector and the state prototype distribution. For abrupt early warning monitoring, abrupt early warning is run in parallel. This is achieved by calculating the fatigue state index (a value ranging from mild to severe fatigue) within a short time window (the most recent 10 seconds). The system calculates the rate of change of a continuous value (weighted by the fatigue state index) for both mild and severe fatigue. If the rate of change exceeds a dynamically set critical slope (the fatigue state index rises by more than 0.7 within 10 seconds), it is determined that the fatigue state has risen sharply, triggering an emergency warning. The critical slope is adaptively calibrated based on the operator's fatigue baseline from the previous day to ensure sensitivity to individual physiological differences, thereby marking this sudden event as a higher priority risk event. Once any warning condition is met, a structured warning message is automatically encapsulated and sent. This message is in standard JSON format and contains the following core fields: warning type, trigger timestamp, current fatigue state, duration of state or rate of change value, overall confidence level of this judgment, and a simplified code of the fatigue state trend in the most recent minute. All messages are appended with a unique serial number and are pushed in real time to the terminal display warning module and the cloud management platform via an encrypted link, providing complete and traceable data credentials for immediate intervention and post-event analysis. The terminal display and early warning module decodes the state and predicts the trend of fatigue early warning signals based on the Hidden Markov Model, provides real-time early warning and risk alerts for sudden fatigue events, and displays fatigue early warning signals, attention changes and historical trends in real time, realizing the visualization of fatigue state and real-time trend prediction and early warning. The interactive early warning feedback module is used to receive user feedback, including confirming fatigue status and adjusting early warning thresholds, collaboratively optimizing human-machine monitoring strategies, supporting user interactive feedback, optimizing monitoring strategies, and improving the system's adaptability. The edge-cloud collaborative optimization module, based on the federated learning framework and ant colony optimization algorithm, performs multi-user model aggregation and optimization in the cloud and personalized model tuning on the terminal, realizing dynamic model updates and collaborative optimization across scenarios and users, and enabling cross-user and cross-scenario model collaborative optimization and personalized dynamic updates.

[0021] Example 2, as Figure 1 , Figure 2 As shown, based on Embodiment 1, the present invention provides a technical solution: the terminal display warning module performs the following steps: after receiving the structured warning message, it uses a hidden Markov model to perform state backtracking and trend prediction, draws fatigue change curves and state transition diagrams, realizes the intelligent leap from passive alarm to active prediction, improves the predictability and scientific nature of fatigue management, and displays the current fatigue status, warning level, attention concentration index and historical fatigue trend charts in real time on the terminal interface, providing intuitive and comprehensive information presentation, assisting operators to clearly understand their own status changes and the system's judgment basis, and when a sudden fatigue warning occurs, it activates an audible and visual alarm and pushes risk warning information to prompt users to take intervention measures. Through multi-sensory strong prompts, it ensures that high-risk states are perceived in a timely and effective manner, prompting timely response; It should be noted that upon receiving a structured early warning message conforming to the standard JSON format, the terminal display and warning module immediately initiates the parsing process. This module first extracts the fatigue state sequence data from the message and then calls the deployed Hidden Markov Model (HMM) for state backtracking analysis. Specifically, the model uses the observation sequence of the previous hour (length L, corresponding to 1800 time steps, generated by a sliding window with a step size of 2 seconds) as input, combined with the trained state transition probability matrix A and observation probability matrix B, and uses a forward-backward algorithm to calculate the posterior probability of each time point in the historical sequence belonging to each hidden state (awake, mild fatigue, severe fatigue). Based on this, the module accurately reconstructs the smooth probability distribution curve of fatigue state over any past time period, and utilizes the temporal dependence characteristics of HMM to predict the probability evolution of the state within the next 30 seconds, calculating the predicted probability value of each state, thus extending the function from passive alarm to active trend prediction. After the analysis and prediction tasks are completed, multi-dimensional information is fused and dynamically visualized on the terminal interface. The core area of ​​the interface updates the current fatigue state in real time (displayed by a three-color indicator light: green for alertness, yellow for mild fatigue, and red for severe fatigue) and the duration of the state. The sidebar of the interface simultaneously displays the results of recent EEG theta. The attention concentration index is calculated by fusing the / β band power ratio and the slope of the near-infrared HbO signal. Historical data is used to plot two core curves: one is a fatigue change curve over the past 30 minutes, reconstructed based on the posterior probability of the Hidden Markov Model (HMM); the other is a Sankey diagram of state transitions generated from the state transition sequence, visually displaying the frequency and direction of transitions between different fatigue states. All chart data is refreshed every 1 second and synchronized with the backend data service via WebSocket protocol to ensure real-time display. When the received warning message type field is identified as a sudden warning or a state of three consecutive time slices (default 6 seconds), the warning is displayed. When severe fatigue is detected, a multimodal early warning response is immediately triggered, activating an audible and visual alarm. An 800Hz alert sound is emitted for 3 seconds, while a red warning light flashes on the screen border. Simultaneously, a modal risk warning window pops up in the center of the interface, clearly displaying the warning level, triggering reason, and personalized intervention suggestions generated based on a preset rule base. This window can only be closed after the user manually clicks to confirm. The system automatically records the user's response time. The warning event, triggering conditions, system response, and user operation are all recorded in detail, and after being timestamped, they are sent back to the cloud management platform, forming a complete data loop of early warning, intervention, and feedback. The interactive early warning feedback module performs the following steps: providing a user interface for users to confirm or deny the fatigue state currently determined by the system, and supporting manual adjustment of early warning sensitivity parameters to achieve rapid verification and instant optimization of the system's judgment, significantly improving the human-computer interaction efficiency and user subjective satisfaction of the monitoring system; collecting user feedback data and current environmental information to construct a feedback dataset for model optimization and early warning rule calibration; establishing a high-quality labeled dataset with scene context to provide reliable data support for continuous and accurate optimization of models and rules; and dynamically adjusting the early warning threshold and state judgment rules based on feedback data to achieve human-computer collaborative fatigue monitoring strategy optimization, enabling the system to adaptively learn user habits and environmental changes, and continuously improve monitoring accuracy and early warning timeliness. It should be noted that the system provides an interactive feedback panel on the terminal interface. This panel is displayed when an alert is triggered or when the user actively calls it out. The core area of ​​the panel provides two main operation buttons: "Confirm Fatigue" and "Deny Fatigue," allowing users to directly provide feedback on the fatigue status currently determined by the system. The sidebar of the panel provides an alert sensitivity adjustment slider, which corresponds to a continuously adjustable sensitivity coefficient P, with an adjustment range of 0.5 to 2.0 and a default value of 1.0. Users adjust the P value by sliding the slider, and this value will linearly affect the core threshold in the alert decision engine: specifically, the continuous time slice threshold N for persistent alerts will be adjusted to N / P, and the critical slope threshold K for abrupt alerts will be adjusted to K*P. All user interactions are responded to instantly, and the adjusted parameters take effect immediately in subsequent monitoring of the current session. The interface synchronously displays the adjusted threshold value for user confirmation. While recording direct user feedback (confirmation / denial) and parameter adjustment actions (sensitivity coefficient P), the system simultaneously collects multimodal data and environmental context information within a time window before and after the feedback trigger moment to construct structured feedback data samples. Each sample contains the following data layers: the core layer consists of raw signal segments (EEG 256Hz, near-infrared 10Hz) for 30 seconds before and after the feedback trigger moment and the corresponding 10-dimensional normalized feature sequence extracted by the system; the label layer consists of binary labels for user confirmation or denial. The system includes a signature and a user-adjusted sensitivity coefficient P; the context layer contains the current task identifier, ambient light intensity (collected by the terminal's ambient light sensor), and time points (categorized into weekday daytime, weekday nighttime, and rest days). All samples are appended with a globally unique timestamp and a user-anonymous identifier, encrypted locally, and incrementally synchronized to the dedicated feedback database of the cloud management platform, forming a high-quality, context-annotated feedback dataset for continuous model and rule optimization; the cloud management platform periodically analyzes the accumulated feedback dataset to drive iterative optimization of the model and rules. The optimization process consists of two parallel paths: one is the adjustment of early warning rules, where the platform calculates... The system first confirms the success rate of the original rules in fatigue samples and the false alarm rate in negative fatigue samples, and then automatically calibrates the global default threshold parameters in the early warning decision engine accordingly. Secondly, the model parameters are fine-tuned. Confirmed fatigue samples and their contextual features are used as new training data, mixed with the original training set, and incremental learning or periodic retraining is employed to fine-tune the feature extraction layer parameters of the Temporal Convolutional Network (TCN) and the observation probability matrix B of the Hidden Markov Model (HMM) to better adapt to the new characteristic patterns of the current user group. After verification, the optimized rule parameters and model weights are securely distributed to each terminal through the edge-cloud collaborative optimization module, completing a human-machine collaborative optimization loop. The edge-cloud collaborative optimization module performs the following steps: It securely aggregates multi-user local models on the cloud using a federated learning framework to generate a global fatigue recognition model, protecting user data privacy. It updates the model locally on multiple distributed terminals, aggregating collective knowledge to improve global model performance while ensuring that users' original physiological data is not uploaded or leaked. It then uses an ant colony optimization algorithm to fine-tune key parameters in the global fatigue recognition model, improving its generalization ability and recognition accuracy in different scenarios. This effectively avoids the tediousness and subjectivity of traditional manual parameter tuning, enabling the model to adapt to different working environments and individual differences, achieving better overall recognition results. Finally, it distributes the optimized global fatigue recognition model to each terminal, combining user local data and feedback for personalized fine-tuning, realizing a dynamic and adaptive cross-user, cross-scenario model update mechanism. This generates personalized versions tailored to individual user physiological characteristics and usage habits, achieving precise monitoring with a "personalized experience for every user." It should be noted that a federated learning server is deployed on the cloud management platform to coordinate collaborative training among various terminals. At the start of each training round, the server distributes the current global model weights to the online terminals. Each terminal uses locally stored, de-identified historical data for local training. The local training cycle is set to 5 rounds, employing stochastic gradient descent with a batch size of 32 and a learning rate of 0.001. After training, the terminal calculates the difference between the local model weights and the received global weights, i.e., the model update amount. This update amount is processed using differential privacy technology, perturbing it with noise conforming to a Gaussian distribution (mean 0, standard deviation σ=0.01), before being encrypted and uploaded to the server. The server uses FedA... The VG algorithm securely aggregates all received encrypted updates by calculating their weighted average. Weights are allocated based on the amount of local data on each terminal. After aggregation, the server decrypts the aggregated model update and applies it to the previous global model, generating a new generation of global fatigue recognition model. This process ensures that the original user data never leaves the local terminal, effectively protecting data privacy. An ant colony optimization algorithm is introduced to automatically tune key hyperparameters of the model to improve the generalization performance of the aggregated global model. A set of parameters to be optimized is defined, including the Dropout rate of the temporal convolutional network (search range 0.05 to 0.3) and the state transition probability matrix of the hidden Markov model. The smoothing coefficient (search range 0.1 to 1.0) and the confidence threshold of the early warning decision engine (search range 0.65 to 0.85) were used to initialize 50 artificial ants. Each ant represented a set of random parameter combinations. In each iteration, each ant used its parameter configuration to evaluate the global model performance on a cross-scenario validation set (containing anonymized feature data from various task scenarios such as office, driving, and monitoring) stored in the cloud. The objective function was the weighted harmonic mean of comprehensive recognition accuracy and false alarm rate. Based on the performance evaluation results, the ants left pheromones along the parameter paths they traversed. The better the performance, the greater the pheromone increment. Subsequent ants then used pheromone concentration and heuristic information (parameter priors) to evaluate the model performance. The model selects a path with a certain probability, gradually converging to the optimal parameter region. After 100 iterations, the optimal parameter combination is selected for the final configuration of the global model. The new generation of global fatigue recognition model, after federated learning aggregation and ant colony optimization, is converted into a terminal-deployable format and digitally signed to ensure integrity. It is then distributed to each online terminal in batches through an encrypted channel. After receiving the model, the terminal verifies the signature in the background and loads the new global fatigue recognition model. To achieve personalized adaptation, the terminal then initiates a lightweight fine-tuning process: using the current user's recent (last 7 days) local feedback data (positive samples confirmed by the user through the interactive interface) at an extremely low learning rate (0.0001) Fine-tuning of the parameters of the last layer or the last two layers of the global fatigue recognition model for no more than 3 epochs is performed. This process incorporates the user-adjusted sensitivity coefficient β in real time, using it as a modulating factor for feature input. After fine-tuning, the new model immediately and seamlessly replaces the old model for real-time monitoring. The system establishes a version management mechanism; if performance degrades on the local validation set after personalized fine-tuning, it automatically rolls back to the issued global model version, ensuring system robustness. This forms a dynamic, secure, and continuously evolving closed loop that balances collective intelligence and individual differences.

[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0023] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A dynamic monitoring system for brain fatigue state based on multimodal signal time series analysis, comprising a cloud management platform, characterized in that, The cloud management platform communication connection includes the following modules: Wearable acquisition terminal is used to acquire multimodal EEG and near-infrared signals in real time, and integrates temporal attention mechanism and sliding window technology for adaptive signal enhancement and feature analysis to obtain fatigue signal features; The edge computing module is used to combine temporal convolutional networks and hidden Markov models to perform temporal modeling and real-time identification of fatigue states at the device edge. The terminal display warning module uses a hidden Markov model to decode the state and predict the trend of fatigue warning signals, provides real-time warnings and risk alerts for sudden fatigue events, and displays fatigue warning signals, attention changes and historical trends in real time. An interactive early warning feedback module is used to receive user feedback, including confirming fatigue status and adjusting early warning thresholds, and collaboratively optimizing human-machine monitoring strategies. The edge-cloud collaborative optimization module, based on the federated learning framework and ant colony optimization algorithm, performs multi-user model aggregation and optimization in the cloud and personalized model tuning on the terminal, realizing dynamic model updates and collaborative optimization across scenarios and users.

2. The dynamic monitoring system for brain fatigue state based on multimodal signal time series analysis according to claim 1, characterized in that: The wearable acquisition terminal includes an adaptive anti-interference sensing unit and a local dynamic feature extraction unit. The adaptive anti-interference sensing unit is used to combine deep residual networks and wavelet adaptive noise reduction technology to perform feature analysis on the captured multimodal EEG and near-infrared signals, and suppress motion artifacts and sudden noise in real time. The local dynamic feature extraction unit is used to extract fatigue signal features from multimodal EEG and near-infrared signals by employing sliding window real-time feature extraction technology combined with temporal attention mechanism.

3. The dynamic monitoring system for brain fatigue state based on multimodal signal time series analysis according to claim 2, characterized in that: The adaptive anti-interference sensing unit performs the following steps: Wavelet adaptive denoising technology was used to perform multi-scale decomposition on the captured multimodal raw EEG and near-infrared signals, separating high-frequency noise components and low-frequency physiological signal components. A noise feature recognition model is constructed based on a deep residual network to identify and classify the separated high-frequency noise components, distinguishing motion artifacts, power frequency interference and sudden impulse noise. The filtering parameters are dynamically adjusted based on the noise classification results. An adaptive filter is used to reconstruct the signal, suppress the identified noise components, and output a multimodal physiological signal with improved signal-to-noise ratio.

4. The dynamic monitoring system for brain fatigue state based on multimodal signal time series analysis according to claim 2, characterized in that: The local dynamic feature extraction unit performs the following steps: A sliding window is used to segment the denoised multimodal signal, and the window length and step size are dynamically adjusted according to the task status to achieve real-time streaming processing of the signal. Within each window, a temporal attention mechanism is used to extract weighted features of the power, approximate entropy, and sample entropy of the α, β, and θ bands of the EEG signal, as well as the mean, peak value, slope, and variance of the HbO of the near-infrared signal. The weighted features are spliced ​​and normalized to form a fatigue signal feature sequence.

5. A dynamic monitoring system for brain fatigue state based on multimodal signal time series analysis according to claim 2, characterized in that: The edge computing module includes a time-series dynamic modeling unit and a real-time early warning decision unit; The temporal dynamic modeling unit, based on temporal convolutional networks and hidden Markov models, performs temporal modeling on the fatigue signal features extracted from the wearable acquisition terminal to continuously identify existing fatigue states. The real-time early warning decision unit is used to determine the fatigue state in real time based on the time series modeling results, generate a fatigue early warning signal in combination with early warning rules, and encapsulate it into a structured early warning message.

6. The dynamic monitoring system for brain fatigue state based on multimodal signal time series analysis according to claim 5, characterized in that: The time-series dynamic modeling unit performs the following steps: Temporal convolutional networks are used to extract deep temporal dependencies from the input feature sequences, capturing the dynamic evolution patterns of fatigue states. The output features of the temporal convolutional network are input into the hidden Markov model to establish the state transition probability matrix and observation probability model between fatigue states, and output the observation sequence, where fatigue states include awake, mild fatigue and severe fatigue. The observation sequence is decoded based on the Viterbi algorithm to output the current fatigue state sequence, enabling continuous identification of fatigue state.

7. A dynamic monitoring system for brain fatigue state based on multimodal signal time series analysis according to claim 5, characterized in that: The real-time early warning decision-making unit performs the following steps: Based on the output fatigue state sequence, combined with the preset fatigue threshold and duration rules, it is determined whether the warning condition is triggered. If multiple consecutive time slices are detected to be in a state of severe fatigue, or if the fatigue level rises sharply in a short period of time, a sudden fatigue warning signal is generated. The warning signal, its confidence level, timestamp, and fatigue trend information are encapsulated into a structured warning message and sent to subsequent modules.

8. A dynamic monitoring system for brain fatigue state based on multimodal signal time series analysis according to claim 5, characterized in that: The terminal display warning module performs the following steps: After receiving the structured early warning message, the Hidden Markov Model is used to perform state backtracking and trend prediction, and to draw fatigue change curves and state transition diagrams. The terminal interface displays the current fatigue status, warning level, attention concentration index, and historical fatigue trend charts in real time. When a sudden fatigue warning occurs, an audible and visual alarm will be activated and a risk alert message will be sent to prompt the user to take intervention measures.

9. A dynamic monitoring system for brain fatigue state based on multimodal signal time series analysis according to claim 8, characterized in that: The interactive early warning feedback module performs the following steps: Provides a user interface for users to confirm or deny the fatigue status currently determined by the system, and supports manual adjustment of the warning sensitivity parameters; Collect user feedback data and current environmental information to build a feedback dataset for model optimization and early warning rule tuning; The warning threshold and status determination rules are dynamically adjusted based on feedback data to optimize the human-machine collaborative fatigue monitoring strategy.

10. A dynamic monitoring system for brain fatigue state based on multimodal signal time series analysis according to claim 9, characterized in that: The edge-cloud collaborative optimization module performs the following steps: In the cloud, a federated learning framework is used to securely aggregate multi-user local models to generate a global fatigue recognition model, protecting user data privacy. The ant colony optimization algorithm is used to fine-tune the key parameters in the global fatigue recognition model. The optimized global fatigue recognition model is then distributed to each terminal. Personalized fine-tuning is performed by combining local user data and feedback to achieve a dynamic and adaptive cross-user and cross-scenario model update mechanism.

Images