A multi-modal driving safety monitoring system based on hydrogel electrodes

Abstract

The application discloses a kind of multimodal driving safety monitoring systems based on hydrogel electrode, it is related to driving safety monitoring technical field.The application includes edge device and cloud, wherein edge device includes synchronous acquisition module, real-time monitoring module, safety policy module and channel detection module;Through high-precision data acquisition and synchronization, ensure that each physiological and environmental signal is accurately aligned;From multiple dimensions, capture the change of driver state, timely warning reduces the risk of accident, realize comprehensive real-time monitoring and multidimensional evaluation;Channel failure detection switches standby channel in time, ensure that signal acquisition is continuous and stable;Data management and incremental learning of cloud constantly improve the adaptability of system to new driving scene, prevent model recession;Realize;Continuous self-adaptation and model iteration optimization.In general, the system can not only monitor and warn driving fatigue state in real time, and continuously improve monitoring accuracy and system stability through the closed-loop feedback of edge and cloud, provide guarantee for driving safety.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of driving safety monitoring, in particular to a multi-modal driving safety monitoring system based on hydrogel electrodes. BACKGROUND

[0002] In recent years, traffic accidents occur frequently, and the number of accidents and casualties caused by fatigue driving is high every year; the wearable devices currently used can provide health monitoring for drivers, but are limited to single-mode monitoring and cannot reflect the multi-dimensional physiological characteristics of driving fatigue; in addition, commercial wet electrodes and gel electrodes have problems of complicated operation and short service life in brain electrical signal acquisition, specifically, the commercial wet electrode needs to be coated with conductive gel, which has the risk of skin irritation; the impedance of the gel electrode increases over time, which cannot meet the continuous one-week monitoring requirement, therefore, there is an urgent need for an electrode that can record long-term high-fidelity physiological electrical signals, and the existing fatigue prediction models are mostly black box algorithms, which cannot quantify the contribution of each physiological characteristic to the fatigue state. SUMMARY

[0003] In view of the deficiencies of the prior art, the present application provides a multi-modal driving safety monitoring system based on hydrogel electrodes, which solves the problems raised in the background art.

[0004] To achieve the above purpose, the present application realizes the technical scheme as follows: a multi-modal driving safety monitoring system based on hydrogel electrodes, comprising an edge device and a cloud, wherein the edge device comprises a synchronous acquisition module, a real-time monitoring module, a safety strategy module and a channel detection module;

[0005] The synchronous acquisition module is connected in communication with each sensor to acquire physiological electrical signals and environmental parameters in real time, and performs time synchronization processing on the physiological electrical signals and environmental parameters to eliminate the phase difference between the channels; wherein the physiological electrical signals include ECG signals, EMG signals, EEG signals and EOG signals; the environmental parameters include steering wheel angle and vehicle speed;

[0006] The real-time monitoring module performs real-time safety monitoring on the driver based on the acquired physiological electrical signals and extracts features to output driving feature parameters; wherein the driving feature parameters include the R θα of the EEG signal, the SDNN of the ECG signal, the f close of the EOG signal, the RMS value of the EMG signal, the steering wheel angle variance σ 2 , and the vehicle speed as an auxiliary parameter;

[0007] The safety strategy module constructs a multi-modal data fusion model based on the driving feature parameters to determine whether the driver is in fatigue driving, and accordingly triggers the corresponding fatigue warning and executes the corresponding safety strategy; sends each fatigue warning and its corresponding original data to the cloud as a warning log.

[0008] a channel detection module, which analyzes and fits the impedance values of the acquisition channels of each physiological signal to determine whether the channels have failure risks and switches the channels accordingly;

[0009] The cloud receives the early warning logs of each edge device, and performs data management and model iteration on the early warning logs.

[0010] Further, the execution process of the synchronous acquisition module is:

[0011] An analog-to-digital converter controlled by an FPGA is used to assign a unified clock source to all bioelectric channels, eliminate the phase difference between channels, implant PTP on the embedded end, and align the environmental parameters and physiological electrical signals through time stamping.

[0012] Each channel is set to have a frequency of f s = 1 kHz, and a time stamp error of Δt, which requires Δt < 1 ms; a synchronization error analysis is performed to calculate the actual time stamp error where Δt clock represents the FPGA clock jitter, and Δt PTP represents the software synchronization error caused by the precise time protocol; when the actual time stamp error Δt tatal < 1 ms, the multi-modal signal alignment requirement is met, and the real-time safety monitoring module is executed, otherwise the convergence condition is not met, and the time stamp of each channel is adjusted again until the actual time stamp error Δt tatal < 1 ms; the specific adjustment method is:

[0013] Clock phase offset compensation: FPGA dynamically adjusts the clock phase of ADC, and the formula is: where T clock = 1 / f clock Δt error represents the time difference between the actual sampling time and the ideal sampling time.

[0014] PTP clock reset: request a new time reference from the master clock and resynchronize.

[0015] Further, based on the fatigue feature extraction of the EEG signal:

[0016] The original EEG signal is band-pass filtered at 0.5-40 Hz, and FIR or IIR filters are used to remove power frequency and high frequency noise; the Welch method is used to integrate in the theta wave and alpha wave intervals respectively, and the respective powers are obtained, and the former is denoted as P θ and the latter is denoted as P α , and the specific calculation formula is The fatigue state index is defined as the power P θThe ratio R between the power P in the alpha wave interval and the power P in the beta wave interval α is: θα R is compared with a ratio threshold value, and if R θα > 0.8, then the brain electrical fatigue state is determined, otherwise the brain electrical wake state is determined. θα

[0017] Further, based on the ECG signal, the fatigue feature is extracted:

[0018] The ECG signal is applied with a 0.5-150 Hz band-pass filter to remove electromyographic interference and baseline drift; the Pan-Tompkins algorithm is used to extract the R peak position, and the adjacent R peak time difference RR i = t i+1 -t i is calculated, where i is any R peak; an ellipse model is fitted with RR i as the horizontal axis and RR i+1 as the vertical axis, and the model expression is: where N is the total number of detected R peaks, is the mean value between adjacent R peaks; SDNN is compared with an SDNN threshold value, and if SDNN is less than the SDNN threshold value, then the ECG fatigue state is determined, otherwise the ECG wake state is determined.

[0019] Further, based on the EOG signal, the eye movement feature is extracted:

[0020] The EOG signal is removed with low-pass to remove high-frequency noise and low-frequency drift, and amplitude threshold and time threshold are set, and each eye closure is identified by peak detection algorithm, and the number of monitored eye closure events in the preset time window is counted, and the eye closure frequency is calculated as eye closure event number / window duration, and is recorded as f close .

[0021] The eye closure frequency is compared with an eye closure frequency threshold value, and if the eye closure frequency is greater than the eye closure frequency threshold value, then the microsleep state is determined, otherwise the eye movement wake state is determined.

[0022] Further, based on the EMG signal, the fatigue feature is extracted:

[0023] For the EMG signal, a band-pass filter is usually used to remove motion artifacts and low-frequency interference, and the root mean square value of the EMG signal is calculated in a preset time window, and the calculation formula is: where M is the total number of discrete sampling points collected in the time window, and j represents the jth sampling point; if (initial baseline value-RMS)>20%, then the electromyographic fatigue state is determined, otherwise the electromyographic wake state is determined.

[0024] ​​Further, the execution process of the safety policy module is:

[0025] The Z-score normalization is used to eliminate the dimensional differences of different features. The normalized modal features are spliced to form a joint vector for subsequent model input. A multi-modal fusion network based on attention mechanism is constructed, in which each input feature is converted into a hidden vector through a fully connected layer, and then the importance of each feature is calculated through an attention layer. The attention weight calculation formula is: Where a and b are feature indexes, ub represents the attention score of the bth feature, which is obtained through network learning and is used to reflect the importance of the feature in the multi-modal fusion process. a u b a represents the attention score of the ath feature, where a is an index that traverses all features; exp(u a ) represents the exponential transformation of the bth feature score, and the denominator is the sum of the exponents of all feature scores, so that the scores of each feature are normalized into a probability distribution form through the Softmax function, and finally the attention weight w a of each feature is obtained.

[0026] The fused feature vector is input into the subsequent fully connected layer, and the output is mapped into the [0, 1] interval through the Sigmoid activation function, representing the fatigue probability formula: Where Vmax is the safe driving speed of the vehicle driving area; if the fatigue probability is in the fatigue fusion judgment interval [P1, P2], then the mild fatigue state is determined, and the first level warning is triggered. The first level warning specifically is: the reminder light at the instrument panel flashes; if the fatigue probability is greater than the upper limit P2 of the fatigue fusion judgment interval [P1, P2], then the moderate fatigue state is determined, and the second level warning is triggered. The second level warning specifically is: seat vibration + voice prompt; if the second level warning is triggered for three consecutive times, then the severe fatigue state is determined, and the third level warning is triggered. The third level warning specifically is: starting automatic driving takeover.

[0027] Further, the marginal contribution of each feature to the model output is quantified:

[0028] The SHAP method is used to calculate the marginal contribution φ a of each feature to the model output, and the calculation formula is: Where S is a feature subset, i.e. any combination of features; A is a feature set, i.e. the set of all features in the model; the feature with the largest marginal contribution is selected as the main fatigue feature, and the other features are sorted in turn.

[0029] Further, the cloud based on the received warning logs of each edge device, data management and model iteration are performed:

[0030] After receiving the early warning log from each edge device, the early warning log is taken as a new data sample to form a continuously updated data set; the features in the early warning log and the corresponding marginal contribution are cleaned and standardized, and at the same time, the actual driving results can be combined to form a training sample for supervised learning;

[0031] The physiological electrical signal is subjected to wavelet transform, low-frequency coefficients and part of high-frequency details are retained, and the compression rate is calculated: The environmental parameters are compressed using the LZMA algorithm;

[0032] The early warning log is stored by driver ID, and Apache Parquet columnar storage is used to optimize batch query efficiency; automatic synchronization is started after the vehicle is turned off, or real-time synchronization is performed when the network bandwidth is greater than 10Mbps, only the newly added data block is uploaded, and the bandwidth occupation is reduced;

[0033] Using the latest early warning log data, the prediction error of the current model on the data is calculated to obtain L current ; the average loss L old of the model on the previous data in a period of time is retained; to smooth the transition of the model, a weighted loss function is used for updating: 7 / 10 represents the weight of the historical data loss L old , and 3 / 10 represents the weight of the current data loss L current ;

[0034] The gradient descent algorithm is used to update the model parameters, so that the new loss L new is minimized, and in the training process, the early warning log data is processed in batches in a small batch mode to improve the convergence efficiency and generalization ability; cross-validation is performed on the cloud using a large data set to evaluate the prediction accuracy and robustness of the updated model, and ensure that the performance of the updated model in actual monitoring is not lower than the predetermined target; after verification, the cloud downloads the updated model parameters to the edge device through the network, so that the real-time monitoring module always uses the latest and optimized model.

[0035] The present application has the following advantages:

[0036] 1. The polyvinyl alcohol (PVA), phosphoric acid, glycerol and glucose syrup are self-crosslinked by heating, without additional chemical crosslinking agent, avoiding the toxicity residue and environmental pollution problems of traditional crosslinking agent, having the advantages of low cost, non-toxicity and high biocompatibility, meeting the green chemistry concept; the hydrogel electrode has high transparency, concealment and self-adhesion, can form a low impedance stable interface with the skin, and still maintains signal monitoring quality under extreme temperature and long-term continuous use, and is suitable for daily wear and monitoring in harsh environment;

[0037] 2. By using FPGA controlled ADC, a unified clock is provided for all bioelectric channels, and the embedded PTP precise time protocol is used to timestamp mark the data, so as to eliminate the phase difference between channels, ensure the synchronization of sampling time, and if the error convergence condition is not met, two compensation methods are used until the error convergence is met; high-quality, time-aligned raw data is provided for the subsequent real-time monitoring module;

[0038] 3. By using high-quality collected physiological signals, real-time fatigue state evaluation is carried out through filtering, feature extraction (such as the theta / alpha ratio of EEG, the SDNN of ECG, the closed-eye frequency of EOG, and the RMS of EMG) and driving state features (steering angle variance), and multi-modal data such as electroencephalogram, electrocardiogram, electrooculogram and electromyogram are fused to capture the fatigue information of the driver from multiple dimensions, so that more comprehensive and accurate monitoring results are provided; the real-time analysis results can timely reflect the changes of the state of the driver, and provide a reliable basis for subsequent early warning, which is helpful to prevent the risk of fatigue driving;

[0039] 4. By Z-score normalization, feature splicing and multi-modal fusion network based on attention mechanism, each physiological and environmental feature is converted into a fatigue probability, and different levels of early warning (such as instrument panel reminder, seat vibration, voice prompt or even automatic driving takeover) are automatically triggered according to the set threshold interval, and SHAP method is used to analyze the marginal contribution of each feature to provide explainability for decision-making;

[0040] 5. By monitoring the contact impedance of each collection channel (hydrogel electrode) in real time, the impedance change rate is calculated through linear regression or moving average, and when the preset threshold is exceeded, the system automatically switches to the standby channel to identify and switch the failed electrode in time, so as to ensure the continuity and stability of signal collection and avoid affecting the overall monitoring effect due to the failure of a single electrode; the design supports a multi-channel array, and the switching time is less than 50ms, effectively ensuring uninterrupted monitoring;

[0041] 6. By receiving the early warning logs and monitoring data uploaded by the edge device, data cleaning, standardization and compression are carried out, and long-term storage is carried out, a continuously updated data set is constructed, based on the features and marginal contributions in the early warning logs, an incremental learning method is used to iteratively update the multi-modal fusion model, and cross-validation is used to ensure that the model accuracy continuously reaches the predetermined target, and the updated model is issued to the edge device to form an online closed-loop feedback;

[0042] The application not only can monitor and early warn the driving fatigue state in real time, but also continuously improves the monitoring accuracy and system stability through closed-loop feedback between the edge and the cloud, thereby providing a strong guarantee for driving safety. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the description of the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0044] Figure 1 The system module connection diagram of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0046] Embodiment one

[0047] Please refer to Figure 1 The present application provides a technical solution: a multi-modal driving safety monitoring system based on hydrogel electrode, including: edge device (driving vehicle) and cloud, wherein the edge device includes a synchronous acquisition module, a real-time monitoring module, a safety policy module and a channel detection module;

[0048] The synchronous acquisition module is connected with the PHGG hydrogel electrode and other special monitoring sensors (Hall sensor, magnetic induction sensor, etc.) to collect physiological electrical signals and environmental parameters. The PHGG hydrogel is prepared with ethylene glycol (PVA), phosphoric acid (H3PO4), glycerin (Glycerinum) and glucose syrup (Glucose) as main raw materials. This hydrogel electrode has good transparency, can self-adhere to human skin, establish a stable interface with skin tissue, has low interface contact impedance, high signal-to-noise ratio, is suitable for harsh environments such as high temperature and low temperature, can be used for a long time (one week), can perform high-quality electrophysiological monitoring, including electrocardiogram (ECG), electromyogram (EMG), electroencephalogram (EEG) and electrooculogram (EOG);

[0049] It needs to be explained that by heating induced polyvinyl alcohol (PVA), phosphoric acid, glycerol and glucose syrup self-crosslinking, without additional chemical crosslinking agent, avoid the toxicity and environmental pollution problems of traditional crosslinking agent, with low cost, non-toxic, high biocompatibility advantage, in line with the concept of green chemistry; the hydrogel electrode has high transparency, concealability, self-adhesion, can form a low impedance stable interface with the skin, still can maintain signal monitoring quality (high signal-to-noise ratio, no performance attenuation) under extreme temperature (40-80℃) and long-term continuous use (one week), suitable for daily wear and harsh environment monitoring;

[0050] The FPGA-controlled analog-to-digital converter (ADC) is used to assign a unified clock source to all bioelectric channels, eliminate the phase difference between channels, and implant PTP (Precision Time Protocol) on the embedded end to align the environmental parameters (steering wheel angle, vehicle speed) and physiological electrical signals through time stamping.

[0051] The frequency of each channel is set to f s = 1 kHz, and the time stamp error is Δt, which requires Δt < 1 ms; the synchronization error is analyzed, and the actual stamp time error is calculated where Δt clock represents the FPGA clock jitter, and Δt PTP represents the software synchronization error caused by the Precision Time Protocol (PTP); when the actual stamp time error Δt tatal < 1 ms, the multi-modal signal alignment requirement is met, and the real-time safety monitoring module is executed, otherwise the convergence condition is not met, and the time stamp of each channel is adjusted again until the actual stamp time error Δt tatal < 1 ms; the specific adjustment method is as follows:

[0052] Clock phase offset compensation, FPGA dynamically adjusts the clock phase of ADC, the formula is: where T clock = 1 / f clock (e.g. 10 ns corresponds to a 100 MHz clock), and Δt error represents the time difference between the actual sampling time and the ideal sampling time.

[0053] PTP from clock reset: request a new time reference from the master clock and resynchronize;

[0054] By using FPGA controlled ADC, a unified clock is provided for all bioelectric channels, and the data is time-stamped by using embedded PTP precise time protocol, so as to eliminate the phase difference between channels and ensure the synchronization of sampling time (the time stamp error Δt < 1 ms); if the error convergence condition is not met, two compensation methods are used until the error convergence is met; and high-quality, time-aligned raw data is provided for the subsequent real-time monitoring module.

[0055] The real-time monitoring module performs real-time safety monitoring analysis on the driver based on the collected physiological electrical signals to output driving feature parameters; the process of safety monitoring analysis is as follows:

[0056] Fatigue feature extraction based on EEG signal:

[0057] The original EEG (electroencephalogram) signal is subjected to 0.5-40 Hz band-pass filtering, and FIR or IIR filter is used to remove power frequency and high-frequency noise; the Welch method (window length = 4 s, overlap 50%) is used to integrate in the theta wave (4-7 Hz) and alpha wave (8-13 Hz) intervals respectively, to obtain the respective power, and the former is denoted as P θ and the latter as P α , and the specific calculation formula is The fatigue state index is defined as the ratio R θ between the power P α in the theta wave interval and the power P θα in the alpha wave interval, and is as follows: The skilled person determines by experiment that the ratio threshold value when fatigued is 0.8, and the wakeful state is 0.6±0.1; R θ / α is compared with the ratio threshold value, if R θ / α > 0.8, it is determined that the EEG is in a fatigued state, otherwise it is determined that the EEG is in a wakeful state;

[0058] Fatigue feature extraction based on ECG signal:

[0059] The ECG (electrocardiogram) signal is subjected to 0.5-150 Hz band-pass filtering to remove electromyographic interference and baseline drift; the Pan-Tompkins algorithm is used to extract the R peak position, and the adjacent R peak time difference RR i = t i+1 -t i is calculated, where i is any R peak; an ellipse model is fitted with RR i as the horizontal axis and RR i+1 as the vertical axis, and the model expression is as follows: where N is the total number of detected R peaks, and is the mean value between adjacent R peaks; The skilled person in the art determines by experiment that the normal state SDNN range is usually 50-100 ms, and the SDNN in the fatigue state can decrease to about 35±5 ms; the SDNN threshold is set to 40 ms, the SDNN is compared with the SDNN threshold, if the SDNN is less than the SDNN threshold, it is determined that the electrocardiogram is in the fatigue state, otherwise it is determined that the electrocardiogram is in the awake state;

[0060] EOG signal-based eye movement feature extraction:

[0061] The EOG (electrooculogram) signal is subjected to low-pass filtering (such as a cutoff frequency of 0.5-15 Hz) to remove high-frequency noise and low-frequency drift, and an amplitude threshold and a time threshold are set (the skilled person determines by experiment that the amplitude threshold is set to 200 μV and the time threshold is set to 1 s), and a peak detection algorithm is used to identify each eye closure (i.e., a continuous low-amplitude event lasting more than 1 second is an eye closure event), the number of eye closure events monitored in a predetermined time window is counted, and the eye closure frequency is calculated as eye closure events / window duration=f close ;

[0062] The eye closure frequency threshold is determined by experiment to be 0.3 Hz, and the eye closure frequency is compared with the eye closure frequency threshold, if the eye closure frequency is greater than the eye closure frequency threshold, it is determined that the driver is in a micro-sleep state, otherwise it is determined that the driver is in an eye movement awake state;

[0063] EMG signal-based fatigue feature extraction:

[0064] The EMG (electromyogram) signal is usually subjected to band-pass filtering (e.g., 20-450 Hz) to remove motion artifacts and low-frequency interference, and the root mean square (RMS) value of the EMG signal is calculated in a predetermined time window, and the calculation formula is: where M is the total number of discrete sampling points collected in the time window, and j represents the jth sampling point; if the initial baseline value-RMS=>20%, it is determined that the driver is in an electromyogram fatigue state, otherwise it is determined that the driver is in an electromyogram awake state;

[0065] The steering wheel angle variance in the predetermined time window is calculated and denoted as σ 2 , and a higher jitter variance indicates a greater possibility of unstable driving state;

[0066] The R θα of the EEG signal, the SDNN of the ECG signal, the f close of the EOG signal, the RMS value of the EMG signal, and the steering wheel angle variance σ 2 are used as driving feature parameters, and the vehicle speed V is used as an auxiliary parameter;

[0067] According to the above electro signal fatigue state judgment, the output fatigue state result is obtained, and the result is visualized and displayed to timely remind the driver to pay attention to driving safety;

[0068] By using high-quality collected physiological signals, real-time evaluation of fatigue state is carried out through filtering, feature extraction (such as θ / α ratio of EEG, SDNN of ECG, closed-eye frequency of ECG, RMS of EMG) and driving state features (steering wheel angle variance), and multi-modal data such as electroencephalogram, electrocardiogram, electrooculogram and electromyogram are fused to capture the fatigue information of the driver from multiple dimensions, so as to provide more comprehensive and accurate monitoring results. The real-time analysis results can timely reflect the changes of the driver state, provide reliable basis for subsequent warning, and help prevent fatigue driving risks.

[0069] The safety policy module constructs a multi-modal data fusion model based on driving feature parameters to determine whether the driver is in fatigue driving, and executes corresponding safety policies accordingly, specifically:

[0070] Z-score normalization is used to eliminate the dimensional differences of different features, and the formula is: Where μ is the mean value of the feature, and σ is the standard deviation. After normalization, each feature parameter obeys the distribution of zero mean and unit variance. The normalized multi-modal features are spliced to form a joint vector (for example, a 5-dimensional vector represents 5 features), which is used for subsequent model input. A multi-modal fusion network based on attention mechanism (AMF-Net) is constructed, in which each input feature is converted into a hidden vector through a fully connected layer, and then the importance of each feature is calculated through an attention layer. The attention weight calculation formula is: Where a and b are feature indices, and ub represents the attention score of the bth feature (or modality), which is obtained through network (such as fully connected layer) learning and is used to reflect the importance of the feature in the multi-modal fusion process. a u b a represents the attention score of the ath feature, where a is an index that traverses all features; exp(u a ) represents the exponential transformation of the bth feature score, and the denominator is the sum of the exponential of all feature scores, so that the scores of each feature are normalized into a probability distribution form through the Softmax function, and finally the attention weight w a of each feature is obtained.

[0071] The fused feature vector is input into the subsequent fully connected layer, and the output is mapped into the [0, 1] interval through the Sigmoid activation function, representing the formula of fatigue probability: Where Vmax is the safe driving speed of the vehicle driving area, usually taking the value of the limit speed of the driving area, according to the formula, the greater the speed, the greater the risk, and the greater the probability of fatigue driving; the fatigue fusion judgment interval [P1, P2] is determined by the present technical personnel through experiments, wherein P1 = 0.7, P2 = 0.8; if the fatigue probability is in the fatigue fusion judgment interval [P1, P2], then the mild fatigue state is determined, and the first level warning is triggered, the first level warning is specifically: the warning light at the instrument panel is controlled to flash to achieve the purpose of reminding the driver; if the fatigue probability is greater than the upper limit P2 in the fatigue fusion judgment interval [P1, P2], then the moderate fatigue state is determined, and the second level warning is triggered, the second level warning is specifically: seat vibration + voice prompt; if the second level warning is triggered for 3 times in succession, then the severe fatigue state is determined, and the third level warning is triggered, the third level warning is specifically: starting automatic driving takeover;

[0072] Using the SHAP (Shapley Additive Explanations) method, the marginal contribution φ of each feature to the model output is calculated a , the calculation formula is: Where S is a feature subset, that is, any combination of features; A is the feature set, that is, the set of all features in the model; example calculation: let the feature set A = {x1, x2, x3}, calculate the marginal contribution value φ of x1 x1 ,

[0073] Through SHAP analysis, the contribution of each feature can be quantified directly, and the feature with the largest marginal contribution is taken as the main fatigue feature; for example, experiments show that the contribution weight of θ / α ratio is about 0.38, and the closed-eye frequency is about 0.16, and other features are in turn sorted;

[0074] Record each warning and the corresponding physiological signals, environmental parameters, and feature marginal contributions as a warning log, and send it to the cloud;

[0075] Through Z-score normalization, feature splicing, and multi-modal fusion network based on attention mechanism (AMF-Net), each physiological and environmental feature is converted into a fatigue probability, and different levels of warning are automatically triggered according to the set threshold interval (such as instrument panel reminder, seat vibration, voice prompt, and even automatic driving takeover), at the same time, the SHAP method is used to analyze the marginal contribution of each feature to provide explainability for decision-making.

[0076] The channel detection module continuously records the contact impedance value of each physiological signal acquisition channel (i.e. each hydrogel electrode) for analysis and fitting to determine whether the channel has a failure risk; specifically:

[0077] Extract the impedance value of each physiological signal acquisition channel, and sort the impedance value according to the time to form the impedance time sequence of each channel, and use linear regression or moving average method to fit the impedance data in a continuous period of time (for example, 1 hour), and calculate the trend of impedance change with time; According to the fitting result, the impedance change rate is calculated, that is, the relative change percentage of the current impedance value and the initial reference value or the average value of the previous period: Where R current represents the current impedance value, R baseline is the reference value recorded previously; the impedance change threshold value = 15% is determined by the experiment of those skilled in the art, the relative change percentage ΔR is compared with the impedance change threshold value, if the relative change percentage > the impedance change threshold value, then the failure risk is accumulated once, when the failure risk appears continuously for 3 times, it is judged that the channel is a failure channel, and the standby channel is switched (8 channel electrode array is designed, which supports dynamic switching to standby channel), the switching time < 50ms;

[0078] By monitoring the contact impedance of each acquisition channel (hydrogel electrode) in real time, calculating the impedance change rate by linear regression or moving average, and automatically switching to the standby channel when the preset threshold is exceeded, the failure electrode is identified and switched in time, ensuring the continuity and stability of signal acquisition, and avoiding the influence of single electrode failure on the overall monitoring effect; Design supports multi-channel array, switching time is less than 50ms, effectively ensures uninterrupted monitoring.

[0079] The cloud end manages the data and iterates the model based on the received early warning logs of each edge device (each driving vehicle), specifically:

[0080] After receiving the early warning logs from each edge device, it is used as a new data sample to form a continuously updated data set; The features in the early warning log and the corresponding marginal contribution are cleaned and standardized, and at the same time, combined with the actual driving results (such as feedback after fatigue intervention), to form a supervised learning training sample; The physiological electrical signal (EEG / ECG / EOG / EMG) adopts wavelet transform (Daubechies 6 wavelet basis, 5 layer decomposition), retains the low frequency coefficient (LL5) and part of the high frequency detail (LH5 / HL5), and the compression rate is calculated: For environmental parameters (steering wheel angle, vehicle speed), use LZMA algorithm (compression rate ≈ 3:1);

[0081] The early warning logs are stored by driver ID, with a retention period of ≥5 years, using Apache Parquet columnar storage to optimize batch query efficiency; Automatically start synchronization after the vehicle is turned off, or real-time synchronization when the network bandwidth > 10Mbps, only upload new data blocks (identify the difference part through RSYNC algorithm), reduce bandwidth occupation;

[0082] Using the latest warning log data, the prediction error (such as cross-entropy loss or mean square error) on these data is calculated by the current model (here, the model refers to the calculation and analysis process of the real-time monitoring module and the security policy module) to obtain L current ; the average loss L old of the model on the previous data in a period of time is retained 7 / 10 represents the weight of the historical data loss L old , and 3 / 10 represents the weight of the current data loss L current . The weights here can be adjusted according to actual conditions, and the purpose is to balance the stability of historical data and the adaptability of new data.

[0083] The gradient descent algorithm (such as SGD or Adam optimizer) is used to update the model parameters, so that the new loss L new is minimized. In the training process, mini-batch processing can be used to process the warning log data in batches, improving the convergence efficiency and generalization ability of the model. Cross-validation is performed on the cloud using a large data set (including historical logs and the latest warning logs) to evaluate the prediction accuracy and robustness of the updated model, ensuring that the performance of the updated model in actual monitoring does not fall below the established target (such as prediction accuracy > 92%). After verification, the cloud will update the model parameters to the edge device through the network, realizing online model iteration, so that the real-time monitoring module always uses the latest and optimized model.

[0084] By receiving the warning logs and monitoring data uploaded by the edge device, data cleaning, standardization, and compression (using wavelet transform, LZMA algorithm, etc.) are performed, and long-term storage is performed. At the same time, a continuously updated data set is constructed, and based on the features and marginal contributions in the warning logs, an incremental learning method (using a weighted loss function and EWC regularization term) is used to iteratively update the multi-modal fusion model, and cross-validation is used to ensure that the model accuracy continuously reaches the predetermined target. The updated model is then distributed to the edge device, forming an online closed-loop feedback.

[0085] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can make equivalent substitutions or changes to the technical solutions and inventive concepts of the present application within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A hydrogel electrode-based multi-modal driving safety monitoring system, comprising an edge device and a cloud, wherein the edge device comprises a synchronous acquisition module, a real-time monitoring module, a safety strategy module and a channel detection module; characterized in that: The synchronous acquisition module is connected in communication with each sensor to acquire physiological electrical signals and environmental parameters in real time, and performs time synchronization processing to eliminate the phase difference between channels; The physiological electrical signals include ECG signals, EMG signals, EEG signals and EOG signals; the environmental parameters include steering wheel angle and vehicle speed; the execution process of the synchronous acquisition module is: An analog-to-digital converter controlled by an FPGA is used to assign a unified clock source to all bioelectric channels to eliminate the phase difference between channels, and a PTP is implanted in the embedded end to align the environmental parameters and physiological electrical signals through time stamping; The frequency of each channel is set to The time stamp error is Δt, and Δt < 1 ms is required. Synchronization error analysis, calculate the actual stamp time error Wherein represents the FPGA clock jitter, represents the software synchronization error caused by the precise time protocol; when the actual stamp time error <1ms, then the multi-modal signal alignment requirement is met, and the real-time safety monitoring module is executed, otherwise the convergence condition is not met, and the adjustment of the time stamp of each channel is re-performed until the actual stamp time error <1ms; the specific adjustment method is: Clock phase offset compensation, FPGA dynamic adjustment ADC uses clock phase, the formula is: Wherein , The time difference between the actual sampling time and the ideal sampling time is represented. The PTP resets from the clock: requests a new time reference from the master clock to resynchronize; The real-time monitoring module performs real-time safety monitoring on the driver based on the acquired physiological electrical signals and feature extraction to output driving feature parameters; The safety strategy module constructs a multi-modal data fusion model based on the driving feature parameters to determine whether the driver is in fatigue driving, and accordingly triggers the corresponding fatigue warning and executes the corresponding safety strategy; The driving characteristic parameters include the SDNN of the ECG signal, the RMS value of the EMG signal, the steering wheel angle variance and the vehicle speed as an auxiliary parameter. Each fatigue warning and its corresponding raw data are sent to the cloud as a warning log; The channel detection module analyzes and fits the impedance value of each physiological signal acquisition channel to determine whether the channel is at risk of failure and accordingly performs channel switching; The cloud receives the warning logs of each edge device, and performs data management and model iteration. Fatigue feature extraction based on EEG signals:

2. A hydrogel electrode based multi-modal driving safety monitoring system according to claim 1, characterized in that, Fatigue feature extraction based on ECG signals: The original EEG signal was bandpass filtered from 0.5 to 40 Hz, and an FIR or IIR filter was used to remove power frequency and high-frequency noise. The Welch method was employed to integrate the signals in the θ and α wave regions respectively, obtaining their respective powers, which were denoted as P. θ and P α The specific calculation formula is as follows: The fatigue state index is defined as the power P in the θ-wave range. θ With power P in the α wave region α The ratio between for: ;Will Compared with the ratio threshold, if If the value is greater than 0.8, it is considered a state of brain fatigue; otherwise, it is considered a state of brain wakefulness.

3. A hydrogel electrode based multi-modal driving safety monitoring system according to claim 2, characterized in that, Eye movement feature extraction based on EOG signals: ECG signal was filtered by 0.5-150 Hz band-pass filter to remove electromyographic interference and baseline drift; Pan-Tompkins algorithm was used to extract R-peak position and calculate the time difference between adjacent R-peaks where i is any one R-peak; the horizontal axis is and the vertical axis is , an elliptical model was fitted, and the model expression is where N is the total number of detected R-peaks, is the mean value between adjacent R-peaks; SDNN was compared with the SDNN threshold value, if SDNN is less than the SDNN threshold value, it is determined that the electrocardio state is fatigue, otherwise it is determined that the electrocardio state is awake.

4. A hydrogel electrode based multi-modal driving safety monitoring system according to claim 3, characterized in that, Compare the closed-eye frequency with the closed-eye frequency threshold value; if the closed-eye frequency is greater than the closed-eye frequency threshold value, it is determined that the driver is in a micro-sleep state, otherwise it is determined that the driver is in an eye movement wake state. Low-pass filter is used to remove high-frequency noise and low-frequency drift of the EOG signal, and amplitude threshold and time threshold are set. Each eye closure is identified by peak detection algorithm. The number of monitored eye closure events in the preset time window is counted, and the eye closure event number / window duration=eye closure frequency, which is recorded as ; Fatigue feature extraction based on EMG signals:

5. A hydrogel electrode based multi-modal driving safety monitoring system according to claim 4, wherein, The execution process of the safety strategy module is: For EMG signal, band-pass filter is usually used to remove motion artifacts and low-frequency interference. The root mean square value of EMG signal is calculated in a preset time window, and the calculation formula is: wherein M is the total number of discrete sampling points collected in the time window, and j represents the jth sampling point; if (initial baseline value-RMS) > 20%, it is judged as muscle fatigue state, otherwise it is judged as muscle clear state.

6. A hydrogel electrode based multi-modal driving safety monitoring system according to claim 5, wherein, Z-score normalization is used to eliminate the dimensional differences of different features, the normalized multi-modal features are spliced to form a joint vector for subsequent model input; a multi-modal fusion network based on attention mechanism is constructed, in which each input feature is converted into a hidden vector through a fully connected layer, and the importance of each feature is calculated through an attention layer; Quantify the marginal contribution of each feature to the model output: The attention weight calculation formula is: wherein a and b are feature indexes, u b represents the attention score of the bth feature, which is obtained through network learning and is used to reflect the importance of the feature in the multi-modal fusion process; u a represents the attention score of the ath feature, wherein a is an index traversing all features; exp(u b） represents that the score of the bth feature is subjected to exponential transformation, and the denominator is the sum of the exponential of all feature scores, so that the scores of the features are normalized into a probability distribution form through a Softmax function, and finally the attention weight w a corresponding to each feature is obtained; The fused feature vector is input to a subsequent fully connected layer, and the output is mapped to the interval [0, 1] through a Sigmoid activation function, representing the formula of fatigue probability: wherein Vmax is the safe driving speed of the vehicle driving area; if the fatigue probability is in the fatigue fusion judgment interval [P1, P2], a mild fatigue state is determined, and a first-level warning is triggered. The first-level warning specifically includes: controlling the reminder light at the instrument panel to flash; if the fatigue probability is greater than the upper limit P2 of the fatigue fusion judgment interval [P1, P2], a moderate fatigue state is determined, and a second-level warning is triggered. The second-level warning specifically includes: seat vibration + voice prompt; if the second-level warning is triggered for three consecutive times, a severe fatigue state is determined, and a third-level warning is triggered. The third-level warning specifically includes: starting automatic driving takeover.

7. A hydrogel electrode based multi-modal driving safety monitoring system according to claim 6, wherein, The cloud performs data management and model iteration based on the received warning logs of each edge device: Using the SHAP method, the marginal contribution of each feature to the model output is calculated , the calculation formula is: , wherein S is a feature subset, that is, any combination of features; A is a feature set, that is, a set of all features in the model; the feature with the largest marginal contribution is taken as the main fatigue feature, and the other features are sequentially sorted.

8. A hydrogel electrode based multi-modal driving safety monitoring system according to claim 7, characterized in that, After receiving the warning logs from each edge device, they are used as new data samples to form a continuously updated data set; the features in the warning logs and the corresponding marginal contributions are cleaned and standardized, and can be combined with the actual driving results to form supervised learning training samples; The warning logs are stored by driver ID, using Apache Parquet columnar storage to optimize batch query efficiency; automatic synchronization is started after the vehicle is turned off, or real-time synchronization is performed when the network bandwidth is > 10Mbps, only new data blocks are uploaded to reduce bandwidth occupation; The physiological electrical signal is subjected to wavelet transform, low frequency coefficients and part of high frequency details are reserved, and compression rate is calculated: The environmental parameters are compressed using the LZMA algorithm. ​ Using the latest warning log data, the prediction error on the data is calculated by the current model to obtain ; Retain the average loss of the model over time on previous data ; to smooth the transition of the model, use a weighted loss function to update: , 7 / 10 represents the weight of the loss of historical data , 3 / 10 represents the weight of the loss of current data ; The model parameters are updated using a gradient descent algorithm, so that the new loss is minimized. In the training process, the early warning log data is processed in batches in a small batch manner, improving the convergence efficiency and generalization ability of the model. Cross-validation is performed on a large data set in the cloud to evaluate the prediction accuracy and robustness of the updated model, ensuring that the performance of the updated model in actual monitoring does not fall below the established target. After verification, the cloud downloads the updated model parameters to the edge device through the network, so that the real-time monitoring module always uses the latest and optimized model.

Images