Multi-mode driving safety monitoring system based on hydrogel electrode

Abstract

The invention discloses a multi-mode driving safety monitoring system based on a hydrogel electrode, and relates to the technical field of driving safety monitoring. The system comprises an edge device and a cloud, wherein the edge device comprises a synchronous acquisition module, a real-time monitoring module, a security policy module and a channel detection module; through high-precision data acquisition and synchronization, each physiological and environmental signal is ensured to be accurately aligned; the state change of the driver is captured from multiple dimensions, early warning is carried out in time, the accident risk is reduced, and comprehensive real-time monitoring and multi-dimensional evaluation are achieved; standby channels are switched in time during channel failure detection, and continuous and stable signal acquisition is ensured; the adaptability of the system to a new driving scene is continuously improved through cloud data management and incremental learning, and model recession is prevented; implementing; and carrying out continuous self-adaption and model iterative optimization. Generally speaking, the system not only can monitor and warn the driving fatigue state in real time, but also can continuously improve the monitoring precision and the system stability through the closed-loop feedback of the edge and the cloud, thereby providing guarantee for the driving safety.

Description

Technical Field

[0001] The present invention relates to the technical field of driving safety monitoring, and specifically to a multi-modal driving safety monitoring system based on hydrogel electrodes. Background Art

[0002] In recent years, traffic accidents have occurred frequently, and the number of accidents and casualties caused by fatigue driving each year remains high. The wearable devices currently in use can provide health monitoring for drivers, but are limited to single-modal monitoring and cannot reflect the multi-dimensional physiological characteristics of driving fatigue. In addition, commercial wet electrodes and gel electrodes have problems such as cumbersome operation and short service life in electroencephalogram (EEG) signal acquisition. Specifically, commercial wet electrodes need to apply conductive gel, which poses a risk of skin irritation. The impedance of gel electrodes increases over time and cannot meet the requirement of continuous monitoring for a week. Therefore, there is an urgent need for electrodes that can record long-term high-fidelity physiological electrical signals at present. Moreover, existing fatigue prediction models are mostly black-box algorithms and cannot quantify the contribution of each physiological characteristic to the fatigue state. Summary of the Invention

[0003] Aiming at the deficiencies of the prior art, the present invention provides a multi-modal driving safety monitoring system based on hydrogel electrodes, which solves the problems raised in the above background art.

[0004] To achieve the above objectives, the present invention is realized through the following technical solutions: A multi-modal driving safety monitoring system based on hydrogel electrodes, including an edge device and a cloud. The edge device includes a synchronous acquisition module, a real-time monitoring module, a safety policy module, and a channel detection module.

[0005] The synchronous acquisition module communicates with each sensor in real time to collect physiological electrical signals and environmental parameters, and performs time synchronization processing on them to eliminate the phase difference between channels. The physiological electrical signals include electrocardiogram (ECG) signals, electromyogram (EMG) signals, electroencephalogram (EEG) signals, and electrooculogram (EOG) signals. The environmental parameters include steering wheel angle and vehicle speed.

[0006] The real-time monitoring module performs real-time safety monitoring and feature extraction on the driver based on the collected physiological electrical signals to output driving feature parameters. The driving feature parameters include R of the EEG signal θα , SDNN of the ECG signal, f of the EOG signal close , RMS value of the EMG signal, variance σ of the steering wheel angle 2 , and uses the vehicle speed as an auxiliary parameter.

[0007] The safety policy module constructs a multi-modal data fusion model based on the driving feature parameters to determine whether the driver is fatigued, and triggers corresponding fatigue warnings and executes corresponding safety policies accordingly. Each fatigue warning and its corresponding original data are sent to the cloud as a warning log.

[0008] The channel detection module analyzes and fits based on the impedance values of the acquisition channels of each physiological signal to determine whether there is a risk of channel failure and switches channels accordingly;

[0009] The early warning logs of each edge device received by the cloud are used for data management and model iteration.

[0010] Furthermore, the execution process of the synchronous acquisition module is as follows:

[0011] An analog-to-digital converter controlled by an FPGA is used to allocate a unified clock source for all bioelectric channels, eliminate the phase difference between channels, implant PTP at the embedded end, and align the environmental parameters and physiological electrical signals through timestamps;

[0012] Set the frequency of each channel to f s = 1 kHz, the timestamp error is Δt, and it is required that Δt < 1 ms; Synchronization error analysis, calculate the actual timestamp error where Δt clock represents the FPGA clock jitter, and Δt PTP represents the software synchronization error caused by the Precision Time Protocol; when the actual timestamp error Δt tatal < 1 ms, the multi-modal signal alignment requirement is met, and the real-time safety monitoring module is executed, otherwise the convergence condition is not met, and the timestamps of each channel are adjusted again until the actual timestamp error Δt tatal < 1 ms; The specific adjustment method is as follows:

[0013] Clock phase offset compensation, the FPGA dynamically adjusts the clock phase of the ADC, and the formula is: where T clock = 1 / f clock Δt error represents the time difference between the actual sampling moment and the ideal sampling moment;

[0014] PTP slave clock reset: Request a new time reference from the master clock and resynchronize.

[0015] Furthermore, fatigue feature extraction based on EEG signals:

[0016] Perform 0.5 - 40 Hz band-pass filtering on the original EEG signal, and use FIR or IIR filters to remove power frequency and high-frequency noise; Using the Welch method, integrate in the θ-wave and α-wave intervals respectively to obtain their respective powers, and denote them as P θ and P α , and the specific calculation formula is Define the fatigue state index as the power P in the θ-wave interval θThe ratio R between the power P in the alpha wave interval α is as follows: θα Compare R with the ratio threshold. If R θα > 0.8, it is determined to be an EEG fatigue state; otherwise, it is determined to be an EEG wakeful state. θα

[0017] Further, fatigue feature extraction based on the ECG signal:

[0018] Apply a 0.5 - 150 Hz band - pass filter to the ECG signal to remove electromyogram interference and baseline drift; use the Pan - Tompkins algorithm to extract the R - peak positions and calculate the time difference RR between adjacent R - peaks i = t i+1 - t i where i is any R - peak; with RR i as the abscissa and RR i+1 as the ordinate, fit an ellipse model, and the model expression is: where N is the total number of detected R - peaks, is the mean value between adjacent R - peaks; Compare SDNN with the SDNN threshold. If SDNN is less than the SDNN threshold, it is determined to be an ECG fatigue state; otherwise, it is determined to be an ECG wakeful state.

[0019] Further, eye movement feature extraction based on the EOG signal:

[0020] Use a low - pass filter on the EOG signal to remove high - frequency noise and low - frequency drift, set amplitude and time thresholds, identify each eye - closing event through the peak - detection algorithm, count the number of detected eye - closing events within a preset time window, and the number of eye - closing events / window duration = eye - closing frequency, which is denoted as f close ;

[0021] Compare the eye - closing frequency with the eye - closing frequency threshold. If the eye - closing frequency is greater than the eye - closing frequency threshold, it is judged to be in a microsleep state; otherwise, it is judged to be in an eye - movement wakeful state.

[0022] Further, fatigue feature extraction based on the EMG signal:

[0023] For the EMG signal, usually use a band - pass filter to remove motion artifacts and low - frequency interference, and calculate the root - mean - square value of the EMG signal within a preset time window. The calculation formula is: where M is the total number of discrete sampling points collected within the time window, and j represents the j - th sampling point; if (initial baseline value - RMS)> 20%, it is judged to be in an electromyogram fatigue state; otherwise, it is judged to be in an electromyogram wakeful state.

[0024] ​Furthermore, the execution process of the security policy module is as follows:

[0025] Z-score normalization is used to eliminate the dimensional differences of different features. The normalized features of each modality are concatenated to form a joint vector for subsequent model input. A multimodal fusion network based on the attention mechanism is constructed, in which each input feature is converted into a latent vector through a fully connected layer, and then the importance of each feature is calculated through the attention layer. The attention weight calculation formula is: Where a and b are feature indices, ub represents the attention score of the b-th feature, which is obtained through network learning and is used to reflect the importance of the feature in the multimodal fusion process; u a represents the attention score of the a-th feature, where a is an index that traverses all features; exp(u b ) represents the exponential transformation of the b-th feature score, and the denominator is the exponential sum of all feature scores, so that the scores of each feature are normalized into a probability distribution form through the Softmax function, and finally the attention weight w corresponding to each feature is obtained a ;

[0026] The fused feature vector is input into the subsequent fully connected layer, and after the Sigmoid activation function, the output is mapped to the interval [0,1]. The formula for fatigue probability is: in Vmax is the safe driving speed in the vehicle driving area; if the fatigue probability ∈ the fatigue fusion judgment interval [P1, P2], it is determined to be a mild fatigue state and trigger a first-level warning. The first-level warning is specifically: controlling the reminder light on the instrument panel to flash; if the fatigue probability is greater than the upper limit P2 in the fatigue fusion judgment interval [P1, P2], it is determined to be a moderate fatigue state and trigger a second-level warning. The second-level warning is specifically: seat vibration + voice prompt; if the second-level warning is triggered three times in a row, it is determined to be a severe fatigue state and trigger a third-level warning. The third-level warning is specifically: starting automatic driving takeover.

[0027] Furthermore, we can quantify the marginal contribution of each feature to the model output:

[0028] Using the SHAP method, the marginal contribution φ of each feature to the model output is calculated a , the calculation formula is: Among them, S is the feature subset, that is, any combination of features; A is the full set of features, that is, the set of all features in the model; the feature with the largest marginal contribution is regarded as the main fatigue feature, and the other features are sorted in order.

[0029] Furthermore, the cloud performs data management and model iteration based on the early warning logs received from each edge device:

[0030] After receiving the warning logs from each edge device and using them as new data samples, a continuously updated data set is formed; the features and corresponding marginal contributions in the warning logs are cleaned and standardized, and at the same time, they can be combined with the actual driving results to form the training samples for supervised learning;

[0031] Wavelet transform is applied to the physiological electrical signals, and the low-frequency coefficients and some high-frequency details are retained. The compression ratio is calculated as follows: The LZMA algorithm is used to compress the environmental parameters;

[0032] The warning logs are partitioned and stored by driver ID, and Apache Parquet columnar storage is used to optimize the batch query efficiency; after the vehicle shuts off, synchronization is automatically started, or when the network bandwidth > 10 Mbps, real-time synchronization is performed, and only the newly added data blocks are uploaded to reduce bandwidth occupancy;

[0033] Using the latest warning log data, the prediction error on the data is calculated through the current model to obtain L current ; retain the average loss L of the model on the previous data over a period of time old ; to smooth the transition of the model, a weighted loss function is used for update: 7 / 10 represents the weight of the historical data loss L old , and 3 / 10 represents the weight of the current data loss L current ;

[0034] The gradient descent algorithm is used to update the model parameters to minimize the new loss L new ; during the training process, the warning log data is processed in batches in a mini-batch manner to improve the model convergence efficiency and generalization ability; cross-validation is performed using a large data set in the cloud to evaluate the prediction accuracy and robustness of the updated model, ensuring that the performance of the updated model in actual monitoring is not lower than the established target; after passing the verification, the cloud will send the updated model parameters to the edge device through the network, so that the real-time monitoring module always uses the latest and optimized model.

[0035] The present invention has the following beneficial effects:

[0036] 1. By heating to induce the self-crosslinking of polyvinyl alcohol (PVA), phosphoric acid, glycerol, and glucose syrup, no additional chemical crosslinking agent is required, avoiding the toxicity residue and environmental pollution problems of traditional crosslinking agents, having the advantages of low cost, non-toxicity, and high biocompatibility, and conforming to the concept of green chemistry; the hydrogel electrode has high transparency, concealability, and self-adhesion, can form a low-impedance stable interface with the skin, and still maintains the signal monitoring quality under extreme temperatures and long-term continuous use, and is suitable for daily wear and harsh environment monitoring;

[0037] 2. By using an FPGA-controlled ADC to provide a unified clock for all bioelectric channels, the embedded PTP precision time protocol is used to timestamp the data, eliminating phase differences between channels and ensuring synchronization of sampling moments. If the error convergence conditions are not met, two compensation methods are used until the error converges. This provides high-quality, time-aligned raw data for subsequent real-time monitoring modules.

[0038] 3. By utilizing high-quality collected physiological signals, fatigue status is assessed in real time through filtering, feature extraction (such as EEG's θ / α ratio, ECG's SDNN, EOG's eye closure frequency, and EMG's RMS), and driving status characteristics (steering wheel angle variance). This system integrates multimodal data such as EEG, ECG, EOG, and EMG to capture driver fatigue information from multiple dimensions and provide more comprehensive and accurate monitoring results. Real-time analysis results can promptly reflect changes in the driver's status, providing a reliable basis for subsequent warnings and helping to prevent fatigue driving risks.

[0039] 4. Through Z-score normalization, feature concatenation, and a multimodal fusion network based on an attention mechanism, various physiological and environmental features are converted into fatigue probabilities. Based on set thresholds, different levels of warnings (such as instrument panel reminders, seat vibrations, voice prompts, and even autonomous driving takeover) are automatically triggered. Furthermore, the SHAP method is used to analyze the marginal contribution of each feature, providing explainability for decision-making.

[0040] 5. By monitoring the contact impedance of each acquisition channel (hydrogel electrode) in real time, the impedance change rate is calculated through linear regression or moving average, and when it exceeds the preset threshold, it automatically switches to the backup channel to promptly identify and switch the failed electrode, ensuring the continuity and stability of signal acquisition and avoiding the impact of the overall monitoring effect due to the failure of a single electrode. The design supports multi-channel arrays with a switching time of less than 50ms, effectively ensuring uninterrupted monitoring.

[0041] 6. By receiving warning logs and monitoring data uploaded by edge devices, the system cleans, standardizes, and compresses the data for long-term storage. At the same time, it builds a continuously updated data set. Based on the features and marginal contributions in the warning logs, it uses incremental learning methods to iteratively update the multimodal fusion model. Cross-validation is used to ensure that the model accuracy continues to meet the predetermined target. The updated model is then sent to the edge devices to form an online closed-loop feedback loop.

[0042] This invention can not only monitor and warn of driver fatigue in real time, but also continuously improve monitoring accuracy and system stability through closed-loop feedback between the edge and the cloud, providing strong protection for driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0044] Figure 1 It is a schematic diagram of the connection of the system modules of the present invention. Specific embodiments

[0045] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0046] Embodiment 1

[0047] Please refer to Figure 1 , the present invention provides a technical solution: a multi-modal driving safety monitoring system based on hydrogel electrodes, including: an edge device (the vehicle being driven) and the cloud, where the edge device includes a synchronous acquisition module, a real-time monitoring module, a safety policy module, and a channel detection module;

[0048] The synchronous acquisition module communicates with PHGG hydrogel electrodes and other dedicated monitoring sensors (such as Hall sensors, magnetic induction sensors, etc.) to collect physiological electrical signals and environmental parameters. The PHGG hydrogel is prepared with polyvinyl alcohol (PVA), phosphoric acid (H3PO4), glycerin (Glycerinum), and glucose syrup (Glucose) as the main raw materials; this hydrogel electrode has good transparency, can self-adhere to the human skin, establish a stable interface with the skin tissue, has a low interface contact impedance, a high signal-to-noise ratio, is applicable to harsh environments such as high temperature and low temperature, and can also be used for a long time (one week), and can perform high-quality electrophysiological monitoring, including electrocardiogram (ECG), electromyogram (EMG), electroencephalogram (EEG), and electrooculogram (EOG);

[0049] It should be noted that by heating to induce the self-crosslinking of polyvinyl alcohol (PVA), phosphoric acid, glycerol, and glucose syrup, no additional chemical crosslinking agent is required, avoiding the toxicity and environmental pollution problems of traditional crosslinking agents. It has the advantages of low cost, non-toxicity, and high biocompatibility, conforming to the concept of green chemistry; the hydrogel electrode has high transparency, concealability, and self-adhesion, can form a low-impedance stable interface with the skin, and can still maintain the signal monitoring quality (high signal-to-noise ratio, no performance attenuation) under extreme temperatures (40 - 80 °C) and long-term continuous use (one week), and is suitable for daily wear and monitoring in harsh environments;

[0050] An analog-to-digital converter (ADC) controlled by FPGA is used to allocate a unified clock source for all bioelectric channels, eliminate the phase difference between channels, implant the Precision Time Protocol (PTP) at the embedded end, and align environmental parameters (steering wheel angle, vehicle speed) with physiological electrical signals through timestamps;

[0051] Each channel is set to have a frequency of f s = 1 kHz, and the timestamp error is Δt, with the requirement that Δt < 1 ms; for synchronous error analysis, calculate the actual timestamp error where Δt clock represents the FPGA clock jitter, and Δt PTP represents the software synchronization error caused by the Precision Time Protocol (PTP); when the actual timestamp error Δt tatal < 1 ms, the multi-modal signal alignment requirement is met, and the real-time safety monitoring module is executed. Otherwise, the convergence condition is not met, and the timestamps of each channel are readjusted until the actual timestamp error Δt tatal < 1 ms; the specific adjustment method is as follows:

[0052] Clock phase offset compensation: The FPGA dynamically adjusts the clock phase of the ADC. The formula is: where T clock = 1 / f clock (for example, 10 ns corresponds to a 100 MHz clock), and Δt error represents the time difference between the actual sampling moment and the ideal sampling moment;

[0053] PTP slave clock reset: Request a new time reference from the master clock and resynchronize;

[0054] By adopting an FPGA-controlled ADC, a unified clock is provided for all bioelectric channels, and the data is timestamped using the embedded Precision Time Protocol (PTP) to eliminate the phase difference between channels and ensure sampling moment synchronization (requiring a timestamp error Δt < 1 ms). If the error convergence condition is not met, two compensation methods are adopted until the error converges; high-quality, time-aligned raw data is provided for the subsequent real-time monitoring module.

[0055] The real-time monitoring module performs real-time safety monitoring and analysis on the driver based on the collected physiological electrical signals to output driving characteristic parameters; the process of safety monitoring and analysis is as follows:

[0056] Fatigue feature extraction based on EEG signals:

[0057] The original EEG (electroencephalogram) signal is band-pass filtered at 0.5 - 40 Hz using a FIR or IIR filter to remove power frequency and high-frequency noise; using the Welch method (window length = 4 s, overlap 50%), integration is performed in the θ wave (4 - 7 Hz) and α wave (8 - 13 Hz) intervals respectively to obtain their respective powers, and the former are denoted as P θ and P α , and the specific calculation formula is Define the fatigue state index as the ratio R θ between the power P α in the θ wave interval and the power P θα in the α wave interval: Those skilled in the art determine through experiments that the ratio threshold during fatigue is 0.8, and in the waking state is 0.6 ± 0.1; compare R θ / α with the ratio threshold. If R θ / α > 0.8, it is determined to be in an EEG fatigue state, otherwise it is determined to be in an EEG waking state;

[0058] Fatigue feature extraction based on ECG signals:

[0059] Apply a 0.5 - 150 Hz band-pass filter to the ECG (electrocardiogram) signal to remove electromyogram interference and baseline drift; use the Pan-Tompkins algorithm to extract the R peak positions and calculate the time difference RR i = t i+1 - t i between adjacent R peaks, where i is any R peak; with RR i as the horizontal axis and RR i+1 as the vertical axis, fit an ellipse model, and the model expression is: where N is the total number of detected R peaks, is the mean between adjacent R peaks; Those skilled in the art determine through experiments that the normal state SDNN range is usually between 50–100 ms, while the SDNN in the fatigued state may drop to about 35±5 ms; set the SDNN threshold to 40 ms, compare SDNN with the SDNN threshold, if SDNN is less than the SDNN threshold, it is determined as the electrocardiogram fatigued state, otherwise it is determined as the electrocardiogram awake state;

[0060] Eye movement feature extraction based on EOG signals:

[0061] For EOG (electrooculogram) signals, low-pass filtering (such as a cut-off frequency of 0.5–15 Hz) is used to remove high-frequency noise and low-frequency drift. Set the amplitude threshold and time threshold (those skilled in the art determine through experiments that the amplitude threshold is set to 200 μV and the time threshold is set to 1 s). Identify each eye closure through the peak detection algorithm (that is, a continuous low-amplitude duration exceeding 1 second is one eye closure event). Count the number of detected eye closure events within a preset time window. The number of eye closure events / window duration = eye closure frequency, and record it as f close ;

[0062] Determine the eye closure frequency threshold to be 0.3 Hz through experiments, compare the eye closure frequency with it, if the eye closure frequency is greater than the eye closure frequency threshold, it is judged as the microsleep state, otherwise it is judged as the eye movement awake state;

[0063] Fatigue feature extraction based on EMG signals:

[0064] For EMG (electromyogram) signals, band-pass filtering (such as 20–450 Hz) is usually used to remove motion artifacts and low-frequency interference. Calculate the root mean square (RMS) value of the EMG signal within a preset time window. The calculation formula is: where M is the total number of discrete sampling points collected within the time window, and j represents the jth sampling point; if the initial baseline value - RMS => 20%, it is judged as the electromyogram fatigued state, otherwise it is judged as the electromyogram awake state;

[0065] Calculate the variance of the steering wheel angle within a preset time window and record it as σ 2 , and the higher the jitter variance, the greater the possibility of an unstable driving state;

[0066] Take the R of the EEG signal θα 、the SDNN of the ECG signal, the f of the EOG signal close 、the RMS value of the EMG signal, the steering wheel angle variance σ 2 as driving feature parameters, and the vehicle speed V as an auxiliary parameter;

[0067] Based on the fatigue state judgment of the above electrical signals, output the fatigue state result and visualize it to timely remind the driver to pay attention to driving safety;

[0068] By utilizing the physiologic signals collected with high quality, real-time assessment of the fatigue state is carried out through filtering, feature extraction (such as the θ / α ratio of EEG, SDNN of ECG, closed-eye frequency of EOG, RMS of EMG) and driving state features (variance of steering wheel angle). Multimodal data such as electroencephalogram, electrocardiogram, electrooculogram, and electromyogram are fused to capture the driver's fatigue information from multiple dimensions, providing more comprehensive and accurate monitoring results. The real-time analysis results can timely reflect the changes in the driver's state, providing a reliable basis for subsequent early warnings and helping to prevent the risk of fatigue driving.

[0069] The safety policy module constructs a multimodal data fusion model based on driving feature parameters to determine whether the driver is in a state of fatigue driving, and accordingly executes corresponding safety policies. Specifically:

[0070] Z-score normalization is used to eliminate the differences in different feature dimensions. The formula is: where μ is the feature mean and σ is the standard deviation. After normalization, each feature parameter follows a distribution with zero mean and unit variance. The normalized modal features are concatenated to form a joint vector (for example, a 5-dimensional vector represents 5 features) for subsequent model input. A multimodal fusion network based on the attention mechanism (AMF-Net) is constructed. Each input feature is transformed into a hidden vector through a fully connected layer, and then the importance of each feature is calculated through the attention layer. The attention weight calculation formula is: where a and b are feature indices, and ub represents the attention score of the b-th feature (or modality). This score is obtained through learning by the network (such as a fully connected layer) and is used to reflect the importance of this feature in the multimodal fusion process. u a represents the attention score of the a-th feature, where a is an index that traverses all features; exp(u b ) represents the exponential transformation of the score of the b-th feature, and the denominator is the sum of the exponentials of all feature scores. Thus, the scores of each feature are normalized into a probability distribution form through the Softmax function, and finally the attention weight w a corresponding to each feature is obtained;

[0071] The fused feature vector is input into the subsequent fully connected layer. After passing through the Sigmoid activation function, the output is mapped to the interval [0,1]. The formula for representing the fatigue probability is: where Vmax is the safe driving speed in the vehicle's driving area, which is usually taken as the speed limit of the driving area. It can be seen from the formula that the greater the speed, the greater the danger and the greater the probability of fatigue driving. The technicians determined the fatigue fusion judgment interval [P1, P2] through experiments, where P1 = 0.7 and P2 = 0.8. If the fatigue probability ∈ the fatigue fusion judgment interval [P1, P2], it is determined to be a mild fatigue state and trigger a first-level warning. The first-level warning is specifically: controlling the reminder light on the instrument panel to flash to achieve the purpose of reminding the driver. If the fatigue probability is greater than the upper limit P2 in the fatigue fusion judgment interval [P1, P2], it is determined to be a moderate fatigue state and trigger a second-level warning. The second-level warning is specifically: seat vibration + voice prompt; if the second-level warning is triggered three times in a row, it is determined to be a severe fatigue state and trigger a third-level warning. The third-level warning is specifically: starting automatic driving takeover.

[0072] Using the SHAP (Shapley Additive Explanations) method, the marginal contribution φ of each feature to the model output is calculated. a , the calculation formula is: Where S is a feature subset, that is, any combination of features; A is the full feature set, that is, the set of all features in the model; Example calculation: Let the full feature set A = {x1, x2, x3}, calculate the marginal contribution value φ of x1 x1 ,

[0073] Through SHAP analysis, the contribution of each feature can be intuitively quantified, and the feature with the largest marginal contribution can be regarded as the main fatigue feature. For example, experiments show that the contribution weight of the θ / α ratio reaches about 0.38, the eye closure frequency is about 0.16, and other features are ranked in order.

[0074] Record each warning and its corresponding physiological electrical signals, environmental parameters, and marginal contributions as a warning log, and send it to the cloud.

[0075] Through Z-score normalization, feature splicing, and an attention-based multimodal fusion network (AMF-Net), various physiological and environmental characteristics are converted into fatigue probabilities. Different levels of warnings (such as instrument panel reminders, seat vibrations, voice prompts, and even autonomous driving takeover) are automatically triggered according to the set threshold range. At the same time, the SHAP method is used to analyze the marginal contribution of each feature to provide explainability for decision-making.

[0076] The channel detection module continuously records the contact impedance value of each physiological signal acquisition channel (i.e., each hydrogel electrode) for analysis and fitting to determine whether there is a risk of channel failure; specifically:

[0077] Extract the borrowing impedance values of each physiological signal acquisition channel, sort them according to the time of the impedance values to form the impedance time series of each channel, and use methods such as linear regression or moving average to fit the impedance data for a continuous period of time (e.g., 1 hour) to calculate the trend of impedance change over time; calculate the impedance change rate according to the fitting result, that is, the relative change percentage of the current impedance value with respect to the initial reference value or the average value of the previous time period: where R current represents the current impedance value, and R baseline is the previously recorded reference value; those skilled in the art experimentally determine that the impedance change threshold = 15%. Compare the relative change percentage ΔR with the impedance change threshold. If the relative change percentage > the impedance change threshold, then accumulate a failure risk once. When there are 3 consecutive failure risks, then determine that this channel is a failed channel and switch to the backup channel (design an 8-channel electrode array, support dynamic switching to the backup channel), and the switching time < 50 ms;

[0078] By real-time monitoring the contact impedance of each acquisition channel (hydrogel electrode), calculating the impedance change rate through linear regression or moving average, and automatically switching to the backup channel when exceeding the preset threshold to timely identify and switch the failed electrode, ensuring the continuity and stability of signal acquisition, and avoiding affecting the overall monitoring effect due to the failure of a single electrode; design a multi-channel array support with a switching time lower than 50 ms to effectively ensure that the monitoring is not interrupted.

[0079] Based on the warning logs received from each edge device (each driving vehicle), the cloud performs data management and model iteration on them, specifically:

[0080] After receiving the warning logs from each edge device, and using them as new data samples to form a continuously updated data set; clean and standardize the features and corresponding marginal contributions in the warning logs, and at the same time, it can be combined with the actual driving results (such as the feedback after fatigue intervention) to form the training samples for supervised learning; perform wavelet transform (Daubechies 6 wavelet basis, 5-layer decomposition) on the physiological electrical signals (EEG / ECG / EOG / EMG), retain the low-frequency coefficients (LL5) and some high-frequency details (LH5 / HL5), and calculate the compression ratio: For environmental parameters (steering wheel angle, vehicle speed), use the LZMA algorithm (compression ratio ≈ 3:1);

[0081] Partition and store the warning logs according to the driver ID, with a retention period ≥ 5 years, use Apache Parquet columnar storage to optimize the batch query efficiency; automatically start synchronization after the vehicle shuts down, or perform real-time synchronization when the network bandwidth > 10 Mbps, and only upload the newly added data blocks (identify the different parts through the RSYNC algorithm) to reduce bandwidth occupancy;

[0082] Using the latest warning log data, the prediction error (such as cross entropy loss or mean square error) on these data is calculated through the current model (the model here refers to the calculation and analysis process of the above-mentioned real-time monitoring module and security policy module) to obtain L current ; Keep the average loss L of the model on the previous data over a period of time old ; To smooth the transition of the model, a weighted loss function is used for update: 7 / 10 means historical data loss L old The weight of 3 / 10 represents the current data loss L current The weight here can be adjusted according to the actual situation, with the aim of taking into account the stability of historical data and the adaptability of new data;

[0083] Use the gradient descent algorithm (such as SGD or Adam optimizer) to update the model parameters so that the new loss L new Minimize the training process by processing the warning log data in mini-batches to improve the model's convergence efficiency and generalization ability. A large data set (including historical logs and the latest warning logs) is used in the cloud for cross-validation to evaluate the prediction accuracy and robustness of the updated model, ensuring that the updated model performs no worse than the established target in actual monitoring (e.g., prediction accuracy > 92%). After verification, the cloud sends the updated model parameters to the edge device via the network to implement online model iteration, ensuring that the real-time monitoring module always uses the latest and optimized model.

[0084] By receiving the early warning logs and monitoring data uploaded by the edge devices, the data is cleaned, standardized, and compressed (using wavelet transform, LZMA algorithm, etc.) for long-term storage. At the same time, a continuously updated data set is constructed. Based on the features and marginal contributions in the early warning logs, an incremental learning method (using weighted loss function and EWC regularization term) is used to iteratively update the multimodal fusion model. Cross-validation is used to ensure that the model accuracy continues to reach the predetermined target. The updated model is then sent to the edge device to form an online closed-loop feedback.

[0085] The above are only preferred specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A multi-modal driving safety monitoring system based on hydrogel electrodes, comprising an edge device and a cloud. The edge device includes a synchronous acquisition module, a real-time monitoring module, a safety policy module, and a channel detection module. It is characterized in that: The synchronous acquisition module communicates with each sensor to collect physiological electrical signals and environmental parameters in real time, and performs time synchronization processing on them to eliminate the phase difference between channels. The physiological electrical signals include ECG signals, EMG signals, EEG signals, and EOG signals; the environmental parameters include steering wheel angle and vehicle speed. The real-time monitoring module performs real-time safety monitoring and feature extraction on the driver based on the collected physiological electrical signals to output driving feature parameters. Among them The driving characteristic parameters include the R of the EEG signal θα , the SDNN of the ECG signal, the f of the EOG signal close , the RMS value of the EMG signal, the variance σ of the steering wheel angle 2 , and the vehicle speed is used as an auxiliary parameter; The safety policy module constructs a multi-modal data fusion model based on the driving feature parameters to judge whether the driver is fatigued, and accordingly triggers corresponding fatigue warnings and executes corresponding safety policies. Send each fatigue warning and its corresponding original data to the cloud as a warning log. The channel detection module analyzes and fits based on the impedance values of the acquisition channels of each physiological signal to judge whether there is a risk of channel failure, and accordingly performs channel switching. The cloud receives the warning logs of each edge device and performs data management and model iteration on them.

2. A multimodal driving safety monitoring system based on a hydrogel electrode according to claim 1, characterized in that, The execution process of the synchronous acquisition module is as follows: An analog-to-digital converter controlled by FPGA is used to allocate a unified clock source for all bioelectric channels, eliminate the phase difference between channels, implant PTP at the embedded end, and align the environmental parameters and physiological electrical signals through timestamps. Set the frequency adopted by each channel to be f s = 1 kHz, and the timestamp error is Δt, where Δt < 1 ms; Synchronization error analysis, calculating the actual timestamp error where Δt clock represents the FPGA clock jitter, and Δt PTP represents the software synchronization error caused by the Precision Time Protocol; when the actual timestamp error Δt tatal < 1 ms, the multi-modal signal alignment requirement is met, and the real-time safety monitoring module is executed; otherwise, the convergence condition is not met, and the timestamps of each channel are readjusted until the actual timestamp error Δt tatal < 1 ms; the specific adjustment method is as follows: Clock phase offset compensation, the FPGA dynamically adjusts the clock phase of the ADC, and the formula is: where T clock = 1 / f clock , Δt error represents the time difference between the actual sampling time and the ideal sampling time; PTP slave clock reset: Request a new time reference from the master clock and resynchronize.

3. A multimodal driving safety monitoring system based on a hydrogel electrode according to claim 2, characterized in that, Fatigue feature extraction based on EEG signals: The original EEG signal is band-pass filtered at 0.5 - 40 Hz, and a FIR or IIR filter is used to remove power frequency and high-frequency noise; using the Welch method, integration is performed respectively in the theta wave and alpha wave intervals to obtain their respective powers, and the former are respectively denoted as P θ and P α , and the specific calculation formula is Define the fatigue state index as the power P θ in the theta wave interval and the power P α in the alpha wave interval, and the ratio R θα between them is:[[]] Compare R θα with the ratio threshold. If R θα > 0.8, it is determined as the EEG fatigue state, otherwise it is determined as the EEG wake state.

4. A multimodal driving safety monitoring system based on a hydrogel electrode according to claim 3, characterized in that, Fatigue feature extraction based on ECG signals: Apply a 0.5–150 Hz band-pass filter to the ECG signal to remove electromyogram interference and baseline drift; use the Pan-Tompkins algorithm to extract the R peak positions and calculate the time difference RR between adjacent R peaks i = t i+1 - t i , where i is any R peak; with RR i as the horizontal axis and RR i+1 as the vertical axis, fit an ellipse model, and the model expression is: where N is the total number of detected R peaks, is the mean between adjacent R peaks; Compare the SDNN with the SDNN threshold. If the SDNN is less than the SDNN threshold, it is determined to be in a state of electrocardiogram fatigue; otherwise, it is determined to be in a state of electrocardiogram wakefulness.

5. A multimodal driving safety monitoring system based on a hydrogel electrode according to claim 4, characterized in that, Eye movement feature extraction based on EOG signals: The EOG signal is processed by a low-pass filter to remove high-frequency noise and low-frequency drift. An amplitude threshold and a time threshold are set. Each eye closure is identified through a peak detection algorithm. The number of detected eye closure events is counted within a preset time window. The eye closure frequency is calculated as the number of eye closure events divided by the window duration, and is denoted as f close ; Compare the eye closure frequency with the eye closure frequency threshold. If the eye closure frequency is greater than the eye closure frequency threshold, it is judged as the micro-sleep state; otherwise, it is judged as the eye movement wake state.

6. A multimodal driving safety monitoring system based on a hydrogel electrode according to claim 5, characterized in that, Fatigue feature extraction based on EMG signals: For EMG signals, band-pass filtering is usually used to remove motion artifacts and low-frequency interference, and the root mean square value of the EMG signal is calculated within a preset time window. The calculation formula is as follows: where M is the total number of discrete sampling points collected within the time window, and j represents the j-th sampling point; if (initial baseline value - RMS) > 20%, it is judged as the myoelectric fatigue state, otherwise it is judged as the myoelectric wake state.

7. A multimodal driving safety monitoring system based on a hydrogel electrode according to claim 6, characterized in that The execution process of the safety policy module is as follows: Use Z-score normalization to eliminate the differences in different feature dimensions, splice the normalized multi-modal features to form a joint vector for subsequent model input; construct a multi-modal fusion network based on the attention mechanism, where each input feature is converted into a hidden vector through a fully connected layer, and then the importance of each feature is calculated through the attention layer. The calculation formula for the attention weight is as follows: where a and b are feature indices, and u b represents the attention score of the b-th feature, which is obtained through network learning and is used to reflect the importance of this feature in the multi-modal fusion process; u a represents the attention score of the a-th feature, where a is an index that traverses all features; exp(u b ) represents the exponential transformation of the score of the b-th feature, and the denominator is the sum of the exponents of all feature scores, so as to normalize the scores of each feature into a probability distribution form through the Softmax function, and finally obtain the attention weight w a ; The fused feature vector is input into the subsequent fully connected layer. After passing through the Sigmoid activation function, the output is mapped to the interval [0, 1]. The formula for representing the fatigue probability is as follows: where Vmax is the safe driving speed in the vehicle driving area. When the fatigue probability is within the fatigue fusion judgment interval [P1, P2], it is determined that the driver is in a mild fatigue state, and a first-level warning is triggered. The specific content of the first-level warning is: controlling the reminder light on the dashboard to flash. When the fatigue probability is greater than the upper limit P2 of the fatigue fusion judgment interval [P1, P2], it is determined that the driver is in a moderate fatigue state, and a second-level warning is triggered. The specific content of the second-level warning is: seat vibration + voice prompt. If the second-level warning is triggered continuously for 3 times, it is determined that the driver is in a severe fatigue state, and a third-level warning is triggered. The specific content of the third-level warning is: activating the automatic driving takeover.

8. A multimodal driving safety monitoring system based on a hydrogel electrode according to claim 7, characterized in that, Quantify the marginal contribution of each feature to the model output: Using the SHAP method, calculate the marginal contribution φ of each feature to the model output a , and the calculation formula is: where S is the feature subset, that is, any combination of features; A is the full set of features, that is, the set of all features in the model; the feature with the largest marginal contribution is used as the main fatigue feature manifestation, and other features are sorted in turn.

9. A multimodal driving safety monitoring system based on a hydrogel electrode according to claim 8, characterized in that, The cloud performs data management and model iteration based on the warning logs received from each edge device: After receiving the warning logs from each edge device, and using them as new data samples to form a continuously updated data set; clean and standardize the features and corresponding marginal contributions in the warning logs, and at the same time, it can be combined with the actual driving results to form the training samples for supervised learning. Wavelet transform is applied to physiological electrical signals, retaining low-frequency coefficients and some high-frequency details. Compression ratio calculation: The LZMA algorithm is used for compressing environmental parameters; Partition and store the warning logs according to the driver ID, use Apache Parquet columnar storage to optimize the batch query efficiency; automatically start synchronization after the vehicle shuts down, or perform real-time synchronization when the network bandwidth > 10Mbps, and only upload the newly added data blocks to reduce bandwidth occupancy. Using the latest early warning log data, the prediction error on the data is calculated by the current model to obtain L current ; retain the average loss L of the model on the previous data over a period of time old ; to smooth the transition of the model, a weighted loss function is used for updating: 7 / 10 represents the weight of the historical data loss L old and 3 / 10 represents the weight of the current data loss L current ; Update the model parameters using the gradient descent algorithm to minimize the new loss L new Minimize new . During the training process, process the warning log data in batches in a mini-batch manner to improve the model convergence efficiency and generalization ability. Conduct cross-validation using a large dataset in the cloud to evaluate the prediction accuracy and robustness of the updated model, ensuring that the performance of the updated model in actual monitoring is not lower than the established target; After verification, the cloud will send the updated model parameters to the edge device via the network, enabling the real-time monitoring module to always use the latest and optimized model.

Images