An intelligent driving takeover early warning method and device based on multi-modal covert perception

Abstract

This invention discloses an intelligent driving takeover warning method and device based on multimodal covert perception. The method uses a flexible triboelectric nanogenerator (TENG) array and a 4D imaging millimeter-wave radar to collect cardiopulmonary activity signals and chest cavity and head micro-motion signals, respectively. By introducing IMU signals collected by an inertial sensor as a reference, adaptive filtering technology is used to filter out vehicle chassis vibration noise and extract pure cardiopulmonary micro-motion signals. A fusion network based on temporal feature extraction and cross-modal attention mechanism is established to output the driver's physiological load. Finally, the physiological load is linked with the complexity of the external traffic scene to generate a takeover readiness (TRL) score, and dynamic graded wake-up and active degrade intervention are performed accordingly. This solution not only solves the problem of easy failure of traditional visual monitoring, but also solves the problems of difficulty in extracting weak physiological signals in the vehicle motion environment, difficulty in unified representation of multimodal information, and difficulty in dynamic quantification of takeover risk.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving, and in particular to an intelligent driving takeover warning method and device based on multimodal covert perception. Background Technology

[0002] In conditional automated driving (Level 3 and above), drivers are allowed to disengage from driving tasks in specific scenarios. However, when the vehicle encounters complex conditions beyond its Operating Design Domain (ODD), the system issues a Takeover Request (TOR), requiring the driver to regain control of the vehicle within a limited timeframe. Currently, driver monitoring systems (DMS) commonly found in mass-produced vehicles primarily rely on infrared or RGB cameras to capture the driver's facial features (such as eyelid closure and yawning frequency).

[0003] However, existing driver status monitoring and takeover warning technologies have many shortcomings. For example, visual perception is prone to failure. Camera-based algorithms are easily affected by complex lighting conditions (such as backlighting, fluctuating brightness in tunnels), wearing sunglasses, heavy makeup, or physical interference from face coverings (such as masks), leading to failure in facial feature extraction. Furthermore, they cannot detect latent physiological disabilities. Most visual solutions can only capture the driver's "obvious behavior" and cannot detect underlying "latent physiological disabilities" such as "blindness" (eyes open but cognitive resources completely depleted) or sudden cardiovascular discomfort. Additionally, contact-based solutions suffer from a poor user experience. Some attempts to introduce medical-grade monitoring require drivers to wear contact-based physiological sensors (such as smartwatches or heart rate monitors), which severely compromises driving comfort and the seamless driving experience, making large-scale deployment in passenger vehicles difficult. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a systematic technical solution with clearly defined coupling relationships and engineering implementation paths. This solution not only solves the problem of easy failure in traditional visual monitoring but also addresses the difficulties in extracting weak physiological signals in vehicle-mounted motion environments, the challenges in uniformly representing multimodal information, and the difficulty in dynamically quantifying takeover risks. This enables highly robust, covert, and dynamic assessment and intervention of driver takeover capabilities.

[0005] To achieve the above objectives, this invention provides a method for intelligent driving takeover warning based on multimodal covert perception, comprising: acquiring driving information, including vehicle external information and vehicle internal information, wherein the vehicle internal information includes cardiopulmonary activity signals collected by a TENG array stitched inside the driver's seat, chest and head micro-motion signals collected by a 4D imaging millimeter-wave radar, and IMU signals collected by an inertial sensor; using the Z-axis acceleration sequence in the IMU signal as a reference noise input, and adaptively filtering the cardiopulmonary activity signals to obtain reconstructed cardiopulmonary activity signals; extracting a first feature from the reconstructed cardiopulmonary activity signals and extracting a second feature from the chest and head micro-motion signals; inputting the first and second features into a one-dimensional convolutional network for feature fusion through attention weighting, and performing LSTM temporal modeling based on the fused features to output the driver's physiological load index; determining the dynamic traffic scene complexity based on the vehicle external information; determining a takeover readiness score based on the physiological load index, dynamic traffic scene complexity, and stress penalty term; determining the warning and intervention level based on the takeover readiness score, and outputting corresponding control commands.

[0006] Optionally, the cardiopulmonary activity signal, after adaptive filtering and reconstruction, is obtained by using the Z-axis acceleration sequence in the IMU signal as a reference noise input, and includes:

[0007] Assume IMU is The acceleration reference sequence acquired in the axial direction is The construction length is Reference input vector:

[0008] ;

[0009] in, This represents a column vector consisting of the current sampled value and its historical sampled values; the superscript T indicates transpose; t represents time.

[0010] Let the adaptive filter weight vector be:

[0011] ;

[0012] in, This is a column vector of adaptive filter parameters; Indicates the first Each tap coefficient is used to characterize the contribution strength of the corresponding time delay component in the reference noise sequence to the current artifact estimation result;

[0013] The artifact component is then estimated from the vehicle excitation. for: ; the corresponding residual signal Defined as: ;

[0014] in, This represents the raw mixed signal acquired by the TENG array at time t. The raw mixed signal includes the cardiac rhythm impact component caused by heartbeat, the respiratory component, motion artifacts coupled to the seat by vehicle vibration, sensor thermal noise, and random disturbance terms. The residual signal is defined as the reconstructed cardiopulmonary activity signal. ;in, This indicates the reconstructed cardiopulmonary activity signals; It is defined as follows.

[0015] Optionally, the adaptive filter parameter column vector is updated using the following formula:

[0016] ;

[0017] in, Step size factor; To prevent regularization terms with excessively small denominators; This represents the L2 norm.

[0018] Optionally, a first feature is extracted from the reconstructed cardiopulmonary activity signal, including:

[0019] The reconstructed cardiopulmonary activity signal was obtained by short-time Fourier transform:

[0020] ;

[0021] in, Indicates the signal at time. and frequency The short-time Fourier transform result at time t is used to characterize the time-frequency distribution characteristics of the signal within the local time window at time t. This indicates the reconstructed cardiopulmonary activity signals; Indicated by time A window function centered on the current time is used to extract a finite-length segment of signal near the current moment. Represents the integral variable; Represents the imaginary unit;

[0022] Based on the physiological frequency range of the human body at rest, the dominant respiratory frequency band is defined as... Define the dominant cardiac frequency band as The respiratory band energy can be calculated from this. With heartbeat frequency energy They are respectively:

[0023] ;

[0024] ;

[0025] in, This represents the minimum energy level in the respiratory band. This indicates the maximum energy value in the respiratory band. This represents the minimum energy value of the heartbeat frequency band; This represents the maximum energy value of the heartbeat frequency band;

[0026] Constructing statistical eigenvectors of TENG modes :

[0027] ;

[0028] in, and These represent the mean and standard deviation of the cardiac envelope amplitude, respectively. Indicates the interval between adjacent heartbeats; This indicates the rate of change of the cardiac interval; and These represent the power spectrum integrals at low and high frequencies, respectively.

[0029] Optionally, extracting the second feature from the micromotion signals of the chest cavity and head includes:

[0030] Suppose that the phase signal obtained by the radar demodulation within the target range cell is Then its radial displacement relative to the target along the line of sight. satisfy:

[0031] ;

[0032] in, The operating wavelength of the radar; The initial phase constant is given; therefore, the radial displacement is determined based on the phase signal. for:

[0033] ;

[0034] For radial displacement The radial micro-velocity is obtained by performing a first-order difference. :

[0035] ;

[0036] in, The sampling period;

[0037] By combining the spatial compactness of point cloud distribution, the distribution of target scattering centers, and Doppler spectral broadening information, radar mode feature vectors are constructed. :

[0038] ;

[0039] in, Indicates the amplitude of respiratory displacement; Indicates the standard deviation of the micro-motion speed; This indicates the radar's estimated breathing frequency; This indicates the characteristic frequency of high-frequency stress micro-motion; Indicates the energy of the Doppler spectrum; This indicates the compactness of the spatial distribution of point clouds.

[0040] Optionally, the first and second features are input into a one-dimensional convolutional network for feature fusion through attention weighting, and LSTM temporal modeling is performed based on the fused features to output the driver's physiological load index, including:

[0041] Let the length be The actual time span corresponding to the time window is:

[0042] ;

[0043] in, Indicates the time width of the local analysis window; Indicates the sampling frequency;

[0044] Suppose a certain mode is at time 10:00 The input sliding time window is represented as:

[0045] ;

[0046] in, Indicates the current time The corresponding local time window input vector contains a total of 1 consecutive sampling point, corresponding to time 1 At the time Observed values; express A real space of dimension is used to describe the input vector. Depend on It consists of real number elements, and its mathematical form is: 3D real-valued column vector;

[0047] Then the output of the k-th feature channel at time t Defined as:

[0048] ;

[0049] in, Indicates the first The first feature channel within the time window Time-weighted coefficients for each historical sampling point; Indicates the first The bias term for each channel, Represents a non-linear activation function;

[0050] Define the response score for each modality as follows: ;

[0051] ;

[0052] in, and Represents the vector of parameters to be learned; and Indicates time High-level characteristics of the lower TENG mode and radar mode; and These represent the corresponding scores for the TENG mode and the radar mode, respectively. and These represent the TENG mode weights and radar mode weights obtained after normalization mapping, respectively, which are used to characterize the relative contribution of the two modes to the construction of the fused features at the current time.

[0053] The two-modal weights are obtained through normalized mapping: ;

[0054] ;

[0055] Obviously there are:

[0056] ;

[0057] This results in a fused feature vector:

[0058] ;

[0059] To further model the fatigue accumulation, short-term shock, and long-term load evolution processes, latent state variables are introduced. And establish a state recursion relationship:

[0060] ;

[0061] in, This represents the state update function, used to fuse historical states with current fused features; It represents the historical state at time t-1, used to characterize the time series information accumulated in the previous time step; The current fusion feature at time t represents the current input information obtained by weighted fusion of TENG modes and radar modes. Through the above state update process, continuous modeling of the evolution of the driver's physiological state over time can be achieved.

[0062] Finally, the driver's real-time physiological load index is obtained through output mapping. :

[0063] ;

[0064] in, This is the output layer parameter vector; This is a bias term.

[0065] Optionally, the complexity of the dynamic traffic scene is determined based on the vehicle's external information, including:

[0066] Set at time The effective forward sensing area of ​​the vehicle is recorded as follows: A total of [number] were detected in the area. The first dynamic objective, the... The location of the target is Category risk weights are Then the dynamic obstacle density Defined as:

[0067] ;

[0068] in, This indicates the area or volume of the vehicle's effective forward sensing area. Used to differentiate the different contributions of pedestrians, two-wheeled vehicles, and motor vehicles to the risk of takeover;

[0069] Let the confidence level of lane line detection be... The effective visible length is The fitting residual is The reference visible length is The upper limit of the reference residual is Then the recognizability of the lane lines is constructed. for:

[0070] ;

[0071] in, Let be the fusion coefficient, and satisfy:

[0072] ;

[0073] Meteorological disturbance intensity Image contrast degradation rate Visibility degradation index Weather-related disturbance levels and radar effective echo attenuation ratio Joint acquisition, namely:

[0074] ;

[0075] in, , , , Let the weighting coefficients satisfy:

[0076] ;

[0077] Let the first The longitudinal distance between the target and this vehicle is The relative speed is Then the collision time Defined as:

[0078] ;

[0079] in, >0 is a stable term to prevent the denominator from being zero;

[0080] Based on collision time, a single-objective conflict risk function is further defined. :

[0081] ;

[0082] in, This is a time-scale parameter. Therefore, the relative velocity conflict risk at the current moment... Take as:

[0083] ;

[0084] To each , , , Normalization is performed to obtain the corresponding normalized parameters. , , , ;

[0085] Since higher lane line visibility indicates a simpler environment, it is inversely mapped to a risk factor. :

[0086] ;

[0087] This leads to the construction of dynamic traffic scenario complexity:

[0088] ;

[0089] in, , , , Representing dynamic obstacle density Lane line risk items Intensity of meteorological interference and the risk of relative speed conflict In terms of the complexity of dynamic traffic scenarios The weighting coefficients in the equation are used to characterize the relative impact of each risk factor on the complexity of the current traffic scenario.

[0090] Optionally, a takeover readiness score is determined based on the physiological load index, dynamic traffic scenario complexity, and stress penalty items, including:

[0091] Define stress-punishment items as:

[0092] ;

[0093] in, The stress penalty term represents time t; This indicates the magnitude of the change in current heart rate relative to an individual's resting baseline; This is the threshold for sudden changes in heart rate. This is the penalty coefficient; This represents the stress state indicator, which takes a value of 1 when the driver is detected to be in a significant stress state, and a value of 0 otherwise.

[0094] Then the takeover readiness at the current time t Defined as:

[0095] ;

[0096] in, A preference coefficient to adjust the relative influence of physiological load and environmental complexity; Indicates the physiological load index; This indicates the complexity of dynamic traffic scenarios.

[0097] On the other hand, the present invention also provides an intelligent driving takeover warning device based on multimodal covert perception, comprising: an acquisition unit for acquiring driving information, the driving information including vehicle external information and vehicle internal information, wherein the vehicle internal information includes cardiopulmonary activity signals collected by a TENG array sewn into the driver's seat, chest and head micro-motion signals collected by a 4D imaging millimeter-wave radar, and IMU signals collected by an inertial sensor; a preprocessing unit for using the Z-axis acceleration sequence in the IMU signal as a reference noise input, and adaptively filtering the cardiopulmonary activity signal to obtain a reconstructed cardiopulmonary activity signal; and an extraction unit for extracting the reconstructed cardiopulmonary activity signal from the cardiopulmonary activity signal. The system extracts a first feature from the motion signal and a second feature from the micro-motion signals of the chest cavity and head; a first determining unit is used to input the first and second features into a one-dimensional convolutional network for feature fusion through attention weighting, and to perform LSTM temporal modeling based on the fused features to output the driver's physiological load index; a second determining unit is used to determine the dynamic traffic scene complexity based on the vehicle's external information; a third determining unit is used to determine the takeover readiness score based on the physiological load index, the dynamic traffic scene complexity, and the stress penalty item; and an output unit is used to determine the warning and intervention level based on the takeover readiness score and output the corresponding control command.

[0098] The advantages of this invention compared to existing technologies are as follows: By deploying a flexible fabric-based triboelectric nanogenerator array within the driver's seat, weak mechanical coupling signals caused by the driver's cardiopulmonary activity are acquired in a non-contact, concealed, and imperceptible manner; in addition, a 4D imaging millimeter-wave radar is deployed on the top of the cockpit to simultaneously collect the driver's chest breathing fluctuations, head micro-movement trajectories, and Doppler velocity information, making it complementary to the TENG seat signals in terms of physiological rhythm perception and posture micro-movement perception; by introducing vehicle chassis IMU inertial navigation data as an independent reference noise input, the normalized minimum mean square adaptive filtering algorithm is used to estimate and remove vehicle excitation artifacts online, thereby reconstructing a pure physiological mechanical wave signal that can be used for analysis under driving conditions; finally, the denoised multimodal physiological characteristics are jointly modeled with the complexity of the external dynamic traffic scene to form a takeover readiness score, which drives takeover advance adjustment, graded wake-up, and minimum risk maneuver control. Attached Figure Description

[0099] Figure 1 This is a flowchart of an intelligent driving takeover warning method based on multimodal coin perception provided by the present invention;

[0100] Figure 2 This is a simulation diagram of pure cardiopulmonary micro-motion signals provided by the present invention;

[0101] Figure 3This is a structural diagram of an intelligent driving takeover warning device based on multimodal coin perception provided by the present invention. Detailed Implementation

[0102] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0103] Reference Figure 1 This embodiment provides an intelligent driving takeover warning method based on multimodal covert perception, including the following steps:

[0104] S10: Acquire driving information, which includes vehicle external information and vehicle internal information, wherein the vehicle internal information includes cardiopulmonary activity signals collected by the TENG array, chest and head micro-movement signals collected by the 4D imaging millimeter-wave radar, and IMU signals collected by the inertial sensor.

[0105] In this embodiment, a flexible fabric-based triboelectric nanogenerator (TENG) array is stitched into the interior of the driver's seat; a 4D imaging millimeter-wave radar is positioned on the top of the cockpit; and inertial sensors (IMU) and external environment sensing devices (cameras / LiDAR) are installed on the reusable vehicle chassis. Additionally, during the model training phase, a medical-grade polysomnography (PSG) system is used to simultaneously collect data from the real driving environment to form a sample set.

[0106] S20: Using the Z-axis acceleration sequence in the IMU signal as a reference noise input, adaptive filtering is performed on the cardiopulmonary activity signal to obtain the reconstructed cardiopulmonary activity signal.

[0107] Specifically, let's assume the IMU is in The acceleration reference sequence acquired in the axial direction is The construction length is Reference input vector:

[0108] ;

[0109] in, This represents a column vector consisting of the current sampled value and its historical sampled values. Its physical meaning is a reference noise input sequence used to characterize vehicle chassis vibration, road surface disturbance, and changes in vehicle body posture; the superscript T indicates transpose.

[0110] Let the adaptive filter weight vector be:

[0111] ;

[0112] in, For the filter parameter column vector, Indicates the first Each tap coefficient is used to characterize the contribution strength of the corresponding time delay component in the reference noise sequence to the current artifact estimation result.

[0113] The artifact component is then estimated from the vehicle excitation. for:

[0114] ;

[0115] From an engineering perspective, Used to approximate non-physiological motion interference mixed into the original TENG signal.

[0116] Corresponding residual signal Defined as:

[0117] ;

[0118] in, This represents the raw mixed signal acquired by the TENG array at time t.

[0119] In this embodiment, the original mixed signal includes the cardiac rhythm impact component caused by heartbeat, the respiratory component, motion artifacts coupled to the seat by vehicle excitation, sensor thermal noise, and random disturbance terms, as detailed below:

[0120] ;

[0121] in, This represents the cardiac impulse map (BCG) components caused by heartbeats. Indicates respiratory component, This indicates the motion artifacts that are coupled into the seat by vehicle excitation. This represents the sensor's thermal noise and random disturbance terms. The residual signal is defined as the reconstructed cardiopulmonary activity signal: ;in, This indicates the reconstructed cardiopulmonary activity signals; Defined as.

[0122] Since changes in vehicle operating conditions can cause instability in the reference noise amplitude, this invention employs the Normalized Least Mean Square (NLMS) algorithm to adaptively update the weights in order to improve the numerical stability of online updates.

[0123] ;

[0124] in, Step size factor To prevent regularization terms with excessively small denominators, This represents the L2 norm.

[0125] After obtaining the reconstructed cardiopulmonary activity signal Then, its power spectral density Analysis is performed. The low-frequency power spectrum integral related to the autonomic nervous system is defined. and high-frequency power spectrum integral for:

[0126] ;

[0127] ;

[0128] in, , These represent the lower and upper limits of the low-frequency power integration range, respectively. , These represent the lower and upper limits of the high-frequency power integration range, respectively.

[0129] Therefore, the sympathetic nerve activation index was constructed. :

[0130] ;

[0131] Among them, ε>0 represents a stable term. When the sympathetic activation index... Consistently above the individual's historical baseline And exceeds the threshold At this time, it can be determined that the driver is in a state of significant stress, that is:

[0132] ;

[0133] in, As an indicator of stress state, this variable will serve as a penalty triggering factor in the subsequent takeover readiness model.

[0134] Figure 2 This is a simulation diagram of the pure cardiopulmonary dyskinetic (BCG + respiration) signal provided in this embodiment. In the figure, A represents the real physiological signal, B represents the vehicle IMU vibration noise, C represents the original mixed signal collected by TENG, and D represents the physiological signal reconstructed based on NLMS. As can be seen from the figure, the original cardiopulmonary dyskinetic features in the noisy mixed signal are significantly contaminated by vehicle vibration noise, while the physiological signal reconstructed based on the NLMS algorithm can better recover the main waveform features and periodic change patterns of the real signal. This indicates that the present invention can effectively achieve denoising and reconstruction of weak physiological mechanical wave signals under vehicle dynamic conditions, and verifies the technical effect of the present invention in extracting driver cardiopulmonary activity features under complex vibration background.

[0135] S30: Extract a first feature from the reconstructed cardiopulmonary activity signal and extract a second feature from the chest cavity and head micromotion signals.

[0136] Specifically, regarding the reconstructed cardiopulmonary activity signals The following is obtained using the short-time Fourier transform:

[0137] ;

[0138] in, Indicates the signal at time. and frequency The short-time Fourier transform result at time t is used to characterize the time-frequency distribution characteristics of the signal within the local time window at time t. Indicated by time A window function centered at the current time is used to extract a finite-length signal segment near the current moment to achieve local spectral analysis of non-stationary signals. Represents the integral variable; It represents the imaginary unit.

[0139] Based on the physiological frequency range of the human body at rest, the dominant respiratory frequency band is defined as... Define the dominant cardiac frequency band as Therefore, the energy of the respiratory band can be calculated. With heartbeat frequency energy They are respectively:

[0140] ;

[0141] ;

[0142] The physical meanings of the above symbols are explained below: This represents the minimum energy level in the respiratory band. This indicates the maximum energy value in the respiratory band. This represents the minimum energy value of the heartbeat frequency band. This represents the maximum energy value of the heartbeat frequency band.

[0143] Further construct the statistical eigenvectors of the TENG modes :

[0144] ;

[0145] in, and These represent the mean and standard deviation of the cardiac envelope amplitude, respectively. Indicates the interval between adjacent heartbeats. This represents the rate of change of the cardiac interval. and These represent the power spectrum integrals at low and high frequencies, respectively.

[0146] For 4D imaging millimeter-wave radar, this invention employs the FMCW (Frequency-Motion-Controlled Wave) system to detect chest rise and fall and micro-movements of the head. Let the phase signal demodulated by the radar within the target range cell be... Then its radial displacement relative to the target along the line of sight. satisfy:

[0147] ;

[0148] in, For radar operating wavelength, This is the initial phase constant. Therefore, the radial displacement is determined based on the phase signal. for:

[0149] ;

[0150] For radial displacement The radial micro-velocities can be obtained by performing a first-order difference. :

[0151] ;

[0152] in, The sampling period is defined. Further combining the spatial compactness of the point cloud, the distribution of the target scattering centers, and the Doppler spectrum broadening information, radar mode feature vectors are constructed. :

[0153] ;

[0154] in, Indicates the amplitude of respiratory displacement. Indicates the standard deviation of the micro-motion speed. This indicates that the radar estimates the dominant breathing frequency. This represents the characteristic frequency of high-frequency stress micro-motion. Indicates the energy of the Doppler spectrum. This indicates the compactness of the spatial distribution of point clouds.

[0155] In this embodiment, to improve the observability of driver startle, stress, and attention interruption states, this application further performs multi-scale feature calculation on the millimeter-wave radar micro-motion signal. Since chest breathing mainly manifests as low-frequency, large fluctuations, while slight head movements and brief startles are characterized by higher-frequency components, radial displacement... power spectral density Perform integral analysis. Define respiratory bandwidth energy. With micro-motion bandwidth energy They are respectively:

[0156] ;

[0157] ;

[0158] Then a radar stress ratio index can be constructed. :

[0159] ;

[0160] in, To prevent the stability constant from having a denominator of zero. When When the value is significantly higher than the individual baseline, it indicates that the driver may be in a state of sudden startle, tension, stiffness, or abnormal posture adjustment. This indicator will be further involved in the calculation of subsequent physiological penalty items.

[0161] S40: Input the first feature and the second feature into a one-dimensional convolutional network and fuse the features through attention weighting. Then, perform LSTM temporal modeling based on the fused features to output the driver's physiological load index.

[0162] Specifically, to preserve the dynamic characteristics of multimodal physiological signal evolution over time, this invention inputs the TENG signal sequence and the millimeter-wave radar signal sequence into the local temporal feature extraction module using a sliding time window method. If the sampling frequency of each modality is denoted as... Then the length is The actual time span corresponding to the time window is:

[0163] ;

[0164] in, Indicates the time width of the local analysis window. When... When the value is small, the model focuses more on short-term transient changes and is suitable for capturing rapid micro-motion features such as startles and sudden posture disturbances; when When the value is large, the model can cover rhythmic information over a longer time range, making it suitable for characterizing respiratory fluctuations, heart rate cycles, and fatigue accumulation trends. Therefore, Essentially, it reflects the receptive field scale of the local temporal feature extraction module for historical information, and is an important time scale parameter connecting discrete sampling sequences with continuous physiological evolution processes.

[0165] Suppose a certain mode is at time 10:00 The input sliding time window is represented as:

[0166] ;

[0167] in, Indicates the current time The corresponding local time window input vector contains a total of 1 consecutive sampling point, corresponding to time 1 At the time The observed values. Therefore, it can be seen that... The direct physical meaning of is "the number of discrete sample points contained in the local analysis window", while its indirect physical meaning is "the length of historical time covered by the current local feature extraction". express A real space. Used to describe the input vector. Depend on It consists of real number elements, and its mathematical form is: A dimensional real-valued column vector.

[0168] Then the output of the k-th feature channel at time t Defined as:

[0169] ;

[0170] in, Indicates the first The first feature channel within the time window The time-weighted coefficients of each historical sampling point Indicates the first The bias term for each channel, This represents a nonlinear activation function. From the above equation, we can see that... This also determines the number of historical samples participating in the weighted summation, and thus the range of local temporal patterns that the channel can sense. Preferably, The value can be set according to the sampling frequency and the target physiological rhythm range, so that a single time window covers at least one important segment of a complete respiratory cycle, or multiple consecutive heartbeat cycles, thereby taking into account the extraction needs of short-term sudden features and low-frequency rhythm features.

[0171] Considering the varying reliability of different modes under different driving conditions—for example, TENG is better at capturing cardiac disturbances, while radar is better at capturing chest breathing and subtle posture movements—an adaptive feature-level fusion model based on modal response intensity is constructed. First, the response score for each mode is defined as: ;

[0172] ;

[0173] in and The parameter vector to be learned; and Indicates time High-level characteristics of the lower TENG mode and radar mode; and These represent the corresponding scores for the TENG mode and the radar mode, respectively. and These represent the TENG mode weights and radar mode weights obtained after normalization mapping, respectively, which are used to characterize the relative contribution of the two modes to the construction of fused features at the current time.

[0174] The two-modal weights are obtained through normalized mapping: ;

[0175] ;

[0176] Obviously there are:

[0177] ;

[0178] This results in a fused feature vector:

[0179] ;

[0180] To further model the fatigue accumulation, short-term shock, and long-term load evolution processes, latent state variables are introduced. And establish a state recursion relationship:

[0181] ;

[0182] in, This represents the state update function, used to fuse historical states with current fused features; It represents the historical state at time t-1, used to characterize the time series information accumulated in the previous time step; The current fusion feature at time t represents the current input information obtained by weighted fusion of TENG and radar modes. Through the above state update process, continuous modeling of the driver's physiological state evolution over time can be achieved. This expression describes the continuous evolution characteristics of the driver's physiological state over time, without directly relying on fixed gating formulas in existing literature.

[0183] Finally, the driver's real-time physiological load index is obtained through output mapping. :

[0184] ;

[0185] in, This is the output layer parameter vector. This is a bias term. Physiological load index. The higher the value, the closer the driver is to the boundary of fatigue, cognitive dissociation, or physiological incapacity.

[0186] It should be noted that during the training phase, medical-grade polysomnography (PSG) and manually labeled results were used to construct supervisory labels. If the actual workload label is denoted as... Then the regression loss of the model can be defined as:

[0187] ; Where N represents the total number of indicators.

[0188] If the driver's condition is classified simultaneously (normal, deep fatigue, physiological incapacity), the classification loss can be added together to form a joint optimization objective: ;

[0189] in, As a weighting factor, The loss term is for state classification.

[0190] S50: Determine the complexity of the dynamic traffic scenario based on the vehicle's external information.

[0191] It should be noted that takeover capability depends not only on the driver's physiological state but also on the complexity of the current traffic environment. Therefore, this invention further quantifies the external scene perception results into a dynamic traffic complexity index. Let the dynamic obstacle density given by the external environment perception module at time t be... Lane markings are recognizable. The intensity of meteorological interference is The risk of relative speed conflict is It should be noted that dynamic obstacle density, lane line discernibility, weather interference intensity, and relative speed conflict risk are not directly output by a single sensor. Instead, they are upper-level semantic quantities obtained by the vehicle's external environment perception module based on camera, millimeter-wave radar, lidar, and vehicle status information, after detection, tracking, fitting, estimation, and normalization operations.

[0192] Specifically, dynamic obstacle density This is obtained by statistically analyzing the number of dynamically tracked targets within the target area ahead of the vehicle and combining this with the spatial distribution of the targets. Let's assume the time is... The effective forward sensing area of ​​the vehicle is recorded as follows: A total of [number] were detected in the area. The first dynamic objective, the... The location of the target is Category risk weights are Then the dynamic obstacle density It can be defined as:

[0193] ;

[0194] in, This indicates the area or volume of the vehicle's effective forward sensing area. This is used to differentiate the different contributions of pedestrians, two-wheeled vehicles, and motor vehicles to the risk of takeover.

[0195] Lane line visibility Calculated by the lane line detection and fitting module. Let the lane line detection confidence level be... The effective visible length is The fitting residual is The reference visible length is The upper limit of the reference residual is Then it can be constructed as follows:

[0196] ;

[0197] in, Let be the fusion coefficient, and satisfy:

[0198] ;

[0199] Meteorological disturbance intensity Image contrast degradation rate Visibility degradation index Weather-related disturbance levels and radar effective echo attenuation ratio Joint acquisition, namely:

[0200] ;

[0201] in, , , , Let the weighting coefficients satisfy:

[0202] ;

[0203] relative speed conflict risk It is calculated from the relative distance and relative speed between this vehicle and surrounding targets. Let the first... The longitudinal distance between the target and this vehicle is The relative speed is Then the collision time Defined as:

[0204] ;

[0205] in, >0 is a stable term to prevent the denominator from being zero.

[0206] Based on the collision time, a single-objective conflict risk function can be further defined. :

[0207] ;

[0208] in, This is a time-scale parameter. Therefore, the relative velocity conflict risk at the current moment... Possible options:

[0209] ;

[0210] Alternatively, in another implementation, a weighted summation form can also be used:

[0211] ;

[0212] in, For the first Risk weights for each objective.

[0213] To achieve the integration of multiple indicators, the following were respectively conducted: Normalization is performed to obtain the corresponding normalized parameters. , , , ;

[0214] Since higher lane line visibility indicates a simpler environment, it is inversely mapped to a risk factor. :

[0215] ;

[0216] Therefore, a scenario complexity model is constructed:

[0217] ;

[0218] in, , , , Representing dynamic obstacle density Lane line risk items Intensity of meteorological interference and the risk of relative speed conflict In terms of the complexity of dynamic traffic scenarios The weighting coefficients in the equation are used to characterize the relative impact of each risk factor on the complexity of the current traffic scenario.

[0219] S60: Determine the takeover readiness score based on the physiological load index, dynamic traffic scenario complexity, and stress penalty items.

[0220] To form a joint model of takeover readiness level (TRL) with clear monotonicity, this invention incorporates the driver's physiological workload index. With environmental complexity Joint modeling. Considering that both increases should lead to a decrease in takeover capability, and that stress events will introduce additional nonlinear risks, we define:

[0221] ;

[0222] in, The stress penalty term represents time t; This indicates the magnitude of the change in current heart rate relative to an individual's resting baseline. This is the threshold for sudden changes in heart rate. This is the penalty coefficient; This represents a stress state indicator; it takes a value of 1 when a significant stress state is detected in the driver, and a value of 0 otherwise. The takeover readiness level at current time t is then determined. Defined as:

[0223] ;

[0224] in, This is a preference coefficient used to adjust the relative influence of physiological load and environmental complexity. As can be seen from the properties of the exponential function, And it satisfies the following monotonicity:

[0225] ;

[0226] ;

[0227] Therefore, when the driver's physiological workload increases or the complexity of the traffic environment increases, the takeover readiness will decrease significantly; when stress-related penalties are activated, This will further reduce costs rapidly. This form has clear physical meaning, interpretable parameters, and is feasible for engineering implementation.

[0228] In other embodiments, to enhance sensitivity to extreme fatigue boundaries, a fatigue threshold function can be introduced based on the above scheme. ,For example:

[0229] ;

[0230] The corrected takeover readiness It can be written as:

[0231] ;

[0232] in, The fatigue threshold represents the critical boundary by which a driver transitions from a normal physiological state to a high-risk fatigue state. This represents the steepness coefficient of the threshold function. It should be noted that... The optimal values ​​are obtained through statistical analysis of training samples, rather than being arbitrarily set. Specifically, physiological load indices can be based on fatigue label data. The distribution was analyzed, and the group threshold was determined by the optimal discriminant point of the ROC curve; in the individualized implementation phase, the average value of drivers under historical normal conditions could also be used. with standard deviation Make corrections, namely:

[0233] ;

[0234] in, This is the risk amplification factor. Therefore, the fatigue threshold... It can be derived from offline statistical optimization results or combined with individual baselines for online adaptive updates.

[0235] S70: Determine the warning and intervention level based on the takeover readiness score, and output the corresponding control command.

[0236] In this embodiment, based on the aforementioned takeover readiness... Or the revised takeover readiness This invention performs dynamic, tiered intervention under safety time constraints when the system issues a Takeover Request (TOR). Assume the threshold satisfies:

[0237] ;

[0238] The control strategy is then defined as:

[0239] ;

[0240] in:

[0241] Normal state. The driver has a high level of takeover capability, and takeover is only alerted via visual flashing prompts on the HUD;

[0242] Moderate risk status. The driver shows signs of decreased attention or mild fatigue. The system appropriately extends the takeover lead time and adds mild seat vibration or voice prompts.

[0243] : Deep fatigue state. The driver's ability to take over is significantly reduced, and the takeover lead time is reduced from the default value. Extended to (in Prioritize the activation of non-auditory wake-up strategies such as seat belt pretensioning, high-frequency seat vibration, and direct cold air blowing to avoid excessive operation caused by high-pitched alarms;

[0244] Physiological incapacity or very low takeover readiness. If BCG arrest, severe arrhythmia, or other abnormalities are detected... Persistently below The system refuses to relinquish control and directly triggers the Minimum Risk Maneuver (MRM), automatically activating hazard lights and controlling the vehicle to safely decelerate, pull over, or enter the emergency stopping lane.

[0245] To adaptively match the timing of intervention with the current takeover risk, let the default takeover lead time be . Then the dynamic takeover lead time It can be defined as:

[0246] ;

[0247] in, This is the time gain factor. When takeover readiness decreases, the system will automatically issue a takeover request in advance, allowing the driver more time to recover; while when the driver is in good condition, unnecessary premature intervention will not be introduced, thereby reducing false alarms and the burden of human-machine interaction.

[0248] As can be seen from the above embodiments, the intelligent driving takeover method based on multimodal covert perception provided by the present invention deploys a flexible fabric-based triboelectric nanogenerator array in the driver's seat to acquire weak mechanical coupling signals caused by the driver's cardiopulmonary activity in a non-contact, covert, and imperceptible manner; in addition, a 4D imaging millimeter-wave radar is deployed on the top of the cockpit to simultaneously collect the driver's chest breathing fluctuations, head micro-movement trajectory, and Doppler velocity information, making it complementary to the TENG seat signal in terms of physiological rhythm perception and posture micro-movement perception; moreover, the vehicle chassis IMU inertial navigation data is introduced as an independent reference noise input, and the normalized minimum mean square adaptive filtering algorithm is used to estimate and remove vehicle excitation artifacts online, thereby reconstructing a pure physiological mechanical wave signal that can be used for analysis under driving conditions; finally, the denoised multimodal physiological features are jointly modeled with the complexity of the external dynamic traffic scene to form a takeover readiness score, and drive takeover advance adjustment, graded wake-up, and minimum risk maneuver control.

[0249] This solution not only solves the problem of easy failure of traditional visual monitoring, but also solves the problems of difficulty in extracting weak physiological signals in vehicle motion environment, difficulty in unified representation of multimodal information, and difficulty in dynamic quantification of takeover risk, thereby achieving a highly robust, covert, and dynamic assessment and intervention of the driver's takeover ability.

[0250] Reference Figure 3 This embodiment provides an intelligent driving takeover warning device based on multimodal covert perception, comprising:

[0251] The acquisition unit 100 is used to acquire driving information, which includes vehicle external information and vehicle internal information. The vehicle internal information includes cardiopulmonary activity signals collected by the TENG array sewn into the driver's seat, chest and head micro-movement signals collected by 4D imaging millimeter-wave radar, and IMU signals collected by inertial sensors. It should be noted that since the specific acquisition method and process have been described in detail in step S10 of the above-mentioned intelligent driving takeover method based on multimodal covert perception, they will not be repeated here.

[0252] The preprocessing unit 200 is used to take the Z-axis acceleration sequence in the IMU signal as a reference noise input and perform adaptive filtering on the cardiopulmonary activity signal to obtain the reconstructed cardiopulmonary activity signal. It should be noted that since the specific preprocessing method and process have been described in detail in step S20 of the above-mentioned intelligent driving takeover method based on multimodal covert perception, they will not be repeated here.

[0253] Extraction unit 300 is used to extract a first feature from the reconstructed cardiopulmonary activity signal and a second feature from the chest cavity and head micro-motion signal; it should be noted that since the specific extraction method and process have been described in detail in step S30 of the above-mentioned intelligent driving takeover method based on multimodal covert perception, they will not be repeated here.

[0254] The first determining unit 400 is used to input the first feature and the second feature into a one-dimensional convolutional network to perform feature fusion through attention weighting, and to perform LSTM temporal modeling based on the fused features to output the driver's physiological load index. It should be noted that since the specific determining method and process have been described in detail in step S40 of the above-mentioned intelligent driving takeover method based on multimodal covert perception, they will not be repeated here.

[0255] The second determining unit 500 is used to determine the complexity of the dynamic traffic scene based on the vehicle's external information. It should be noted that since the specific determination method and process have been described in detail in step S50 of the above-mentioned intelligent driving takeover method based on multimodal covert perception, they will not be repeated here.

[0256] The third determining unit 600 is used to determine the takeover readiness score based on the physiological load index, dynamic traffic scenario complexity, and stress penalty item. It should be noted that since the specific determination method and process have been described in detail in step S60 of the above-mentioned intelligent driving takeover method based on multimodal covert perception, they will not be repeated here.

[0257] The output unit 700 is used to determine the warning and intervention levels based on the takeover readiness score and output corresponding control commands. It should be noted that the specific output method and process have been described in detail in step S70 of the aforementioned intelligent driving takeover method based on multimodal covert perception, and therefore will not be repeated here.

[0258] In addition, embodiments of the present invention also provide a computer-readable storage medium, wherein the computer-readable storage medium may store a program, which, when executed, includes some or all of the steps of any intelligent driving takeover method based on multimodal covert perception described in the above method embodiments.

[0259] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0260] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0261] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0262] The above description, with reference to the accompanying drawings, illustrates an exemplary flowchart for implementing an intelligent driving takeover method based on multimodal covert perception according to an embodiment of the present invention. It should be noted that the numerous details included in the above description are merely illustrative of the invention and not intended to limit it. In other embodiments of the invention, the method may have more, fewer, or different steps, and the order, inclusion, function, and other relationships between the steps may differ from those described and illustrated.

Claims

1. A method for early warning of intelligent driving takeover based on multimodal covert perception, characterized in that, include: Acquire driving information, which includes vehicle external information and vehicle internal information, wherein the vehicle internal information includes cardiopulmonary activity signals collected by a TENG array sewn into the driver's seat, chest and head micro-movement signals collected by 4D imaging millimeter-wave radar, and IMU signals collected by inertial sensors. Using the Z-axis acceleration sequence in the IMU signal as a reference noise input, the cardiopulmonary activity signal is adaptively filtered to obtain the reconstructed cardiopulmonary activity signal; The first feature is extracted from the reconstructed cardiopulmonary activity signal, and the second feature is extracted from the micromotion signals of the chest cavity and head. The first and second features are input into a one-dimensional convolutional network and fused through attention weighting. Based on the fused features, LSTM temporal modeling is performed to output the driver's physiological load index. The complexity of the dynamic traffic scenario is determined based on the vehicle's external information. The takeover readiness score is determined based on the physiological load index, the complexity of the dynamic traffic scenario, and the stress penalty item. The warning and intervention levels are determined based on the takeover readiness score, and corresponding control commands are output.

2. The intelligent driving takeover warning method according to claim 1, characterized in that, Using the Z-axis acceleration sequence in the IMU signal as a reference noise input, adaptive filtering is performed on the cardiopulmonary activity signal to obtain the reconstructed cardiopulmonary activity signal, including: Assume IMU is The acceleration reference sequence acquired in the axial direction is The construction length is Reference input vector: ； in, This represents a column vector consisting of the current sampled value and its historical sampled values; the superscript T indicates transpose; t represents time. Let the adaptive filter weight vector be: ； in, This is a column vector of adaptive filter parameters; Indicates the first Each tap coefficient is used to characterize the contribution strength of the corresponding time delay component in the reference noise sequence to the current artifact estimation result; The artifact component is then estimated from the vehicle excitation. for: ； Corresponding residual signal Defined as: ； in, This represents the raw mixed signal acquired by the TENG array at time t. The raw mixed signal includes the cardiac rhythm impulse component caused by the heartbeat, the respiratory component, and the vehicle excitation coupled to the seat. Motion artifacts, sensor thermal noise, and random disturbance terms in the data; The residual signal is defined as the reconstructed cardiopulmonary activity signal: ； in, This indicates the reconstructed cardiopulmonary activity signals; It is defined as follows.

3. The intelligent driving takeover warning method according to claim 2, characterized in that, The adaptive filter parameter column vector is updated using the following formula: ； in, Step size factor To prevent regularization terms with excessively small denominators, This represents the L2 norm.

4. The intelligent driving takeover warning method according to claim 1, characterized in that, The first feature is extracted from the reconstructed cardiopulmonary activity signal, including: The reconstructed cardiopulmonary activity signal was obtained by short-time Fourier transform: ； in, Indicates the signal at time. and frequency The short-time Fourier transform result at time t is used to characterize the time-frequency distribution characteristics of the signal within the local time window at time t. This indicates the reconstructed cardiopulmonary activity signals; Indicated by time A window function centered on the current time is used to extract a finite-length segment of signal near the current moment. Represents the integral variable; Represents the imaginary unit; Based on the physiological frequency range of the human body at rest, the dominant respiratory frequency band is defined as... Define the dominant cardiac frequency band as The respiratory band energy can be calculated from this. With heartbeat frequency energy They are respectively: ； ； in, This represents the minimum energy level in the respiratory band. This indicates the maximum energy value in the respiratory band. This represents the minimum energy value of the heartbeat frequency band; This represents the maximum energy value of the heartbeat frequency band; Constructing statistical eigenvectors of TENG modes : ； in, and These represent the mean and standard deviation of the cardiac envelope amplitude, respectively. Indicates the interval between adjacent heartbeats. This represents the rate of change of the cardiac interval. and These represent the power spectrum integrals at low and high frequencies, respectively.

5. The intelligent driving takeover warning method according to claim 1, characterized in that, Extracting the second feature from the micro-motion signals from the chest cavity and head includes: Suppose that the phase signal obtained by the radar demodulation within the target range cell is Then its radial displacement relative to the target along the line of sight. satisfy: ； in, The operating wavelength of the radar; The initial phase constant; Determine the radial displacement based on the phase signal. for: ； For radial displacement The radial micro-velocity is obtained by performing a first-order difference. : ； in, The sampling period; By combining the spatial compactness of point cloud distribution, the distribution of target scattering centers, and Doppler spectral broadening information, radar mode feature vectors are constructed. : ； in, Indicates the amplitude of respiratory displacement; Indicates the standard deviation of the micro-motion speed; This indicates the radar's estimated breathing frequency; This indicates the characteristic frequency of high-frequency stress micro-motion; Indicates the energy of the Doppler spectrum; This indicates the compactness of the spatial distribution of point clouds.

6. The intelligent driving takeover warning method according to claim 1, characterized in that, The first and second features are input into a one-dimensional convolutional network and fused through attention weighting. Based on the fused features, LSTM temporal modeling is performed to output the driver's physiological workload index, including: Let the length be The actual time span corresponding to the time window is: ； in, Indicates the time width of the local analysis window; Indicates the sampling frequency; Suppose a certain mode is at time 10:00 The input sliding time window is represented as: ； in, Indicates the current time The corresponding local time window input vector contains a total of 1 consecutive sampling point, corresponding to time 1 At the time Observed values; express 3D real space; Then the output of the k-th feature channel at time t Defined as: ； in, Indicates the first The first feature channel within the time window Time-weighted coefficients for each historical sampling point; Indicates the first The bias term for each channel; Represents a non-linear activation function; Define the response score for each modality as follows: ; ； in, and Represents the vector of parameters to be learned; and Indicates time High-level characteristics of the lower TENG mode and radar mode; and These represent the corresponding scores for the TENG mode and the radar mode, respectively. and These represent the TENG mode weights and radar mode weights obtained after normalization mapping, respectively, which are used to characterize the relative contribution of the two modes to the construction of the fused features at the current time. The two-modal weights are obtained through normalized mapping: ; ； Obviously there are: ； This results in a fused feature vector: ； To further model the fatigue accumulation, short-term shock, and long-term load evolution processes, latent state variables are introduced. And establish a state recursion relationship: ； in, This represents the state update function, used to fuse historical states with current fused features; It represents the historical state at time t-1, used to characterize the time series information accumulated in the previous time step; The current fusion feature at time t represents the current input information obtained by weighted fusion of TENG modes and radar modes. Finally, the driver's real-time physiological load index is obtained through output mapping. : ； in, This is the output layer parameter vector; This is a bias term.

7. The intelligent driving takeover warning method according to claim 1, characterized in that, The complexity of the dynamic traffic scene is determined based on the vehicle's external information, including: Set at time The effective forward sensing area of ​​the vehicle is recorded as follows: A total of [number] were detected in the area. The first dynamic objective, the... The location of the target is Category risk weights are Then the dynamic obstacle density Defined as: ； in, This indicates the area or volume of the vehicle's effective forward sensing area. Used to differentiate the different contributions of pedestrians, two-wheeled vehicles, and motor vehicles to the risk of takeover; Let the confidence level of lane line detection be... The effective visible length is The fitting residual is The reference visible length is The upper limit of the reference residual is Then the recognizability of the lane lines is constructed. for: ； in, Let be the fusion coefficient, and satisfy: ； Meteorological disturbance intensity Image contrast degradation rate Visibility degradation index Weather-related disturbance levels and radar effective echo attenuation ratio Joint acquisition, namely: ； in, , , , Let the weighting coefficients satisfy: ； Let the first The longitudinal distance between the target and this vehicle is The relative speed is Then the collision time is defined for: ； in, >0 is a stable term to prevent the denominator from being zero; Based on collision time, a single-objective conflict risk function is further defined. : ； in, Given a time-scale parameter, the relative velocity conflict risk at the current moment... Take as: ； To each , , , Normalization is performed to obtain the corresponding normalized parameters. , , , ; Since higher lane line visibility indicates a simpler environment, it is inversely mapped to a risk factor. : ； This leads to the construction of dynamic traffic scenario complexity: ； in, , , , Representing dynamic obstacle density Lane line risk items Intensity of meteorological interference and the risk of relative speed conflict In terms of the complexity of dynamic traffic scenarios The weighting coefficients in the equation are used to characterize the relative impact of each risk factor on the complexity of the current traffic scenario.

8. The intelligent driving takeover warning method according to claim 1, characterized in that, The takeover readiness score is determined based on the physiological load index, dynamic traffic scenario complexity, and stress penalty items, including: Define stress-punishment items as: ； in, The stress penalty term represents time t; This indicates the magnitude of the change in current heart rate relative to an individual's resting baseline; This is the threshold for sudden changes in heart rate. This is the penalty coefficient; This represents the stress state indicator, which takes a value of 1 when a significant stress state is detected in the driver, and a value of 0 otherwise. Then the takeover readiness at the current time t Defined as: ； in, A preference coefficient to adjust the relative influence of physiological load and environmental complexity; Indicates the physiological load index. This indicates the complexity of dynamic traffic scenarios.

9. A smart driving takeover warning device based on multimodal covert perception, characterized in that, include: The acquisition unit is used to acquire driving information, which includes vehicle external information and vehicle internal information. The vehicle internal information includes cardiopulmonary activity signals collected by the TENG array sewn into the driver's seat, chest and head micro-movement signals collected by 4D imaging millimeter-wave radar, and IMU signals collected by inertial sensors. The preprocessing unit is used to take the Z-axis acceleration sequence in the IMU signal as a reference noise input and perform adaptive filtering on the cardiopulmonary activity signal to obtain the reconstructed cardiopulmonary activity signal. An extraction unit is used to extract a first feature from the reconstructed cardiopulmonary activity signal and a second feature from the micromotion signals of the chest cavity and head. The first determining unit is used to input the first feature and the second feature into a one-dimensional convolutional network to fuse the features through attention weighting, and to perform LSTM temporal modeling based on the fused features to output the driver's physiological load index. The second determining unit is used to determine the complexity of the dynamic traffic scene based on the vehicle's external information; The third determining unit is used to determine the takeover readiness score based on the physiological load index, the complexity of the dynamic traffic scenario, and the stress penalty item. The output unit is used to determine the warning and intervention levels based on the takeover readiness score and output the corresponding control commands.

10. A computer-readable storage medium comprising a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent driving takeover warning method based on multimodal covert perception as described in any one of claims 1 to 8.

Images