Multi-mode driver state detection method and system based on residual hybrid gating

Abstract

The invention discloses a multi-modal driver state detection method and system based on residual hybrid gating, and the method comprises the steps: 1, carrying out the synchronous preprocessing of data, and obtaining four paths of standardized time sequence tensors; 2, local-global joint feature extraction: capturing local space-time details in a shallow layer by adopting lightweight convolution, and capturing global dependence by using a normalized DyT-Transform of a replacement layer; obtaining local and global data features; 3, performing residual hybrid gating fusion; and 4, health state classification: performing flattening operation on the fusion features, and outputting the health state probability of the driver through a two-layer full-connection network. According to the method, multi-mode representation rich in semantics and high in robustness is obtained through local-global collaborative modeling and dynamic noise suppression, and the health state of the driver is detected continuously based on the current emotion, pressure, fatigue and distraction states until the current state of the driver is adjusted to the target state. And the driver is ensured to be in a suitable driving state.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, pattern recognition and vehicle driving safety technology, and in particular to a multimodal driver state detection method and system based on residual hybrid gating. Background Technology

[0002] With the rapid development of automotive intelligence and advanced driver assistance systems (ADAS), real-time and reliable driver status monitoring (DMS) has become a core focus of regulations and new vehicle safety assessments. However, current research still has two major limitations. (1) Over-reliance on a single modality, lack of complementarity and redundancy: Visual solutions focus on features such as facial movement units, eyelid closure, or head posture, but their performance drops sharply in low light, sunglasses, or mask scenarios at night; vehicle signal solutions infer the status through steering wheel angular velocity, throttle / brake opening, or longitudinal acceleration, but are easily affected by road conditions, vehicle model differences, and driving style; physiological signal solutions can directly reflect the autonomic nervous system and emotional state, but lead-based acquisition is easily affected by sweat, body movement, and temperature changes, resulting in large signal fluctuations and significant baseline drift, making it difficult to use stably in the long term. (2) Focus on a single health indicator, lack of multi-dimensional comprehensive assessment: Existing systems mostly focus on the single state of "fatigue" or "distraction," often ignoring key dimensions such as emotional abnormalities and stress overload, and are unable to form a global judgment on the driver's health status. To improve robustness and coverage, academia and industry have attempted multimodal fusion. However, early methods only performed feature concatenation or static weighting, lacking explicit differentiation between homogeneous and heterogeneous information. Subsequent methods introduced attention and graph convolution for complementarity, but these generally simplified processing at the decision level, making it difficult to capture temporal context and local-global dependencies. Furthermore, most frameworks still align multimodal features all at once at the network end, failing to suppress noise caused by occlusion or sensor drift in a timely manner at shallow layers, resulting in a high false positive rate. In summary, there is currently a lack of a method that can collaboratively model local-global features within a modality, dynamically suppress noise across modalities, and simultaneously conduct comprehensive evaluation of multidimensional health indicators. Summary of the Invention

[0003] To address the aforementioned issues, this invention proposes a multimodal driver state detection network (LG-RMGNet) based on residual hybrid gating. Unlike traditional methods that only concatenate features at the decision layer or target only a single health state, LG-RMGNet performs deep collaborative modeling both within and across modal levels, enabling comprehensive assessment of multidimensional health states such as emotion, stress, fatigue, and distraction. Specifically, LG-RMGNet uses facial data, physiological signals, vehicle operation data, and environmental road data as inputs to achieve comprehensive, low-latency monitoring of driver health status. To further optimize driver health status in a closed-loop manner, this invention also discloses a personalized comprehensive health navigation system for drivers. This system implements corresponding optimization strategies by real-time detection of the driver's comprehensive health status, achieving a closed-loop process of detection-decision-optimization.

[0004] The technical solution steps of this invention are as follows:

[0005] Step 1: Data synchronization preprocessing;

[0006] Assume the sequences of the four primitive modes are as follows:

[0007]

[0008] in, Represents the original data. , , and These represent the face, electrocardiogram, vehicle, and road, respectively. This represents the time step. Preprocessing includes resampling alignment, noise reduction filtering, and normalization, which can be abstracted into the following formula:

[0009]

[0010] in, Indicates based on common sampling rate Perform resampling alignment. This indicates noise reduction filtering. This represents standardized processing. For the preprocessed data, To standardize sequence length, For the first Number of channels for each modality (Face, ECG, Vehicle, Road).

[0011] Step 2: Local-Global Joint Feature Extraction;

[0012] To simultaneously capture local details (instantaneous facial expressions, minute ECG waveforms, sudden changes in vehicle signals, etc.) and global dependencies (long-cycle heart rate, driving operation modes, etc.), preprocessing is required for each stream of data. Simultaneously capturing local details and global temporal dependencies, and fusing them into a unified-dimensional modal embedding, lays the foundation for subsequent cross-modal fusion. To this end, this invention introduces a two-level encoder of "lightweight CNN + DyT-Transformer" within each modality: a shallow CNN is responsible for local information detection, and a deep DyT-Transformer is responsible for global dependency modeling.

[0013] First, to improve training stability and convergence speed, and enhance robustness against local noise and overfitting, a batch normalization (BN) layer is cascaded after the convolutional layers to reduce gradient explosion / vanishing and stabilize the local feature extraction stage. Second, ReLU is used as the activation function to drive the convolutional channels to learn sparse activation patterns, highlighting local details such as sudden facial expressions and instantaneous ECG fluctuations, thus improving the nonlinear expressive power of shallow networks. Finally, Dropout is used to suppress overfitting of local convolutional branches to a few significant channels and provides lightweight regularization without significantly increasing inference overhead. The local convolutional encoding formula is abstractly represented as follows:

[0014]

[0015] in, This represents two layers of one-dimensional convolution (3 kernels, 64 output channels) + BN + ReLU + Dropout; This indicates the learnable parameters of the convolutional branch; It represents local spatiotemporal details.

[0016] While maintaining the same temporal length, the number of channels in the convolution output is mapped to a unified embedding dimension. This reduces subsequent computation and facilitates cross-modal alignment. The formula is as follows:

[0017]

[0018] in, yes Convolution weights, This indicates that the projected features have channels aligned to the shared dimension. .

[0019] Dynamic-Tanh normalization, compared to layer normalization, can reduce the computational complexity of division and square root operations, thereby reducing latency and memory usage without sacrificing accuracy. The DyT expression is as follows:

[0020]

[0021] in, Represents the input vector (the first one) Frame, channel dimension ), Indicates the learnable amplification factor. Indicates the tangent of the entire curve, with an output range of... , This represents the scaling and translation parameters per channel. This represents the normalized vector.

[0022] A DyT-Transformer global encoder is introduced at the back end of each modality: long-range relationships are explicitly modeled through a self-attention mechanism to reveal high-order semantic dependencies between different time segments, as expressed below:

[0023]

[0024] in, This represents a learnable position vector, which injects positional information into the sequence; Indicates by A network composed of layers of DyT-Transformers, Indicates learnable parameters, It represents global dependency features, which imply long-term temporal sequence and contextual semantics.

[0025] Step 3: Residual hybrid gating fusion;

[0026] The features obtained in step two contain both local and global information but lack cross-modal interaction. This step introduces residual hybrid gating fusion to dynamically suppress noise: when a modality is missing or distorted, its weight is reduced; complementarity is fully utilized: when other modalities provide discriminative information, it is supplemented as needed; and residual paths are preserved: to prevent over-fusion from diluting the main modality features.

[0027] First, cross-attention is used to extract bidirectional cross-modal information, accurately align the temporal context, and explicitly model cross-modal dependencies.

[0028]

[0029] in, Indicates the dominant mode. Indicates auxiliary mode, Indicates "dominant mode" "Auxiliary modality" refers to cross-modal interaction features; its shape remains consistent with the primary modality.

[0030] Secondly, retention factors and update factors are generated through a gated weight generation network, as follows:

[0031]

[0032]

[0033] in, Indicates global average pooling. This indicates a splicing operation between the primary and secondary modes. This represents the weights and biases of the gated weight generation network. represent Function, output , Indicates the retention factor (preserving the original features). This represents the update factor (introducing interactive features).

[0034] Then, residual mixing and fusion are performed as follows:

[0035]

[0036] in, This indicates element-wise multiplication. If the principal mode signal-to-noise ratio is high, then... A high proportion of these modalities means they retain more of their own information. If the auxiliary modality is more reliable, then... It has a higher proportion and incorporates more cross-modal information; It is the result of fusing the main mode with a single auxiliary mode.

[0037] Dominant mode With the other 3 auxiliary paths After fusion, the average is calculated along the path dimension. This averaging operation ensures that the contributions of each path are equal, avoiding the introduction of additional parameters due to relearning the weights. The final cross-modal representation is as follows:

[0038]

[0039] in, The total number of modalities is used to obtain the final fused features from the four channels. .

[0040] Finally, temporal average pooling and multimodal vector concatenation are performed on the fused features to obtain the final fused features, which can be used for later classification, as shown below:

[0041]

[0042] Step 4: Classify health status;

[0043]

[0044] in, Indicates the weight of the category header. Indicates bias. for Predicted probability of a health-like state.

[0045] Thus, multimodal fusion features are mapped to interpretable probability outputs, enabling a comprehensive assessment of the driver's emotions, stress, fatigue, and distraction.

[0046] Personalized integrated health navigation system for drivers;

[0047] To address four health conditions, a personalized comprehensive health navigation system for drivers is introduced. This system uses a closed-loop approach to regulate the driver's physiological and psychological state, ensuring they remain within a suitable driving range. The principle is as follows:

[0048]

[0049] in, The personalized comprehensive health index is divided into four levels within the value range (0-10, where 8-10 indicates excellent health status, 6-7 indicates good health status, 4-5 indicates moderate health status, and 0-3 indicates poor health status). , , and These represent the driver's emotional state, stress, fatigue, and distraction levels, respectively (with values ​​ranging from 0 to 10, where 8-10 indicates excellent health, 6-7 indicates good to moderate health, 4-5 indicates moderate health, and 0-3 indicates poor health). , , and These represent personalized weights for emotion, stress, fatigue, and distraction (ranging from 0 to 1), and satisfy the following conditions: ; This represents the health status level correction (within the range of -10 to 0), and satisfies... Furthermore, the numerical range and real-time correction amount based on the personalized comprehensive health index. Health status optimization measures can be divided into four levels: Level 1 optimization (PCHI=8-9, excellent health status, soothing music), Level 2 optimization (PCHI=6-7, good health status, vibration intervention), Level 3 optimization (PCHI=4-5, medium health status, voice warning), and Level 4 optimization (PCHI=0-3, poor health status, emergency stop). Attached Figure Description

[0050] Figure 1 The present invention relates to a multimodal driver state detection method and system based on residual hybrid gating.

[0051] Figure 2 Generate a network graph for the gating weights.

[0052] Figure 3 This is a residual hybrid fusion structure diagram (the facial modality is the reference benchmark, and the other modalities are similar). Detailed Implementation

[0053] The proposed multimodal driver state detection method and system based on residual hybrid gating is implemented using the PyTorch deep learning framework and runs on an Nvidia GeForce RTX 4090 GPU with 24GB of memory. During training, the Adam optimizer is used with a batch size of 64. The model is trained for 50 epochs with an initial learning rate of 0.00001. Cross-entropy loss is chosen as the loss function, suitable for multi-class classification problems. To capture temporal dependencies and meet the model's input format requirements, a sliding window strategy is used to augment the data, with a 5-second window for sample partitioning. This implementation compares four network versions (CNN_Transformer, CNN_Transformer_CSA, CNN_Transformer_Gate, and CNN_Transformer_RMG) and four input configurations (Face; Face+ECG; Face+ECG+Vehicle; Face+ECG+Vehicle+Road) on a self-built CDHD dataset. The results are shown in Table 1. The experimental results show that the dual evolution of multimodal information and fusion mechanism significantly improves the detection performance: when ECG, Vehicle and Road signals are gradually introduced on the basis of a single Face modality, the accuracy of the four tasks all show a step-like increase. Among them, ECG has the most significant gain for stress and distraction recognition, Vehicle behavioral features further strengthen stress and fatigue discrimination, and Road scene information provides the final leap in distraction detection. At the same time, the model structure is gradually expanded from the basic CNN-Transformer to introduce cross-scale attention (CSA), channel gating (Gate) and even residual hybrid gating (RMG). The refinement of the fusion strategy also brings steady improvement. In particular, RMG effectively suppresses heterogeneous noise and strengthens the mutual information of key modalities by applying dynamic weights to the residual branches. As a result, the accuracy of emotion, stress, fatigue and distraction under the four-modal configuration reaches 87.86%, 91.91%, 98.13% and 69.22%, respectively, which is 2.70 and 1.31 percentage points higher than the baseline in stress and distraction tasks, respectively. In summary, the horizontal expansion of the input dimension provides complementary information sources, while the vertical deepening of the fusion mechanism determines whether the information can be fully integrated. CNN_Transformer_RMG achieved the best results in all four driver state detection tasks in the four-modal scheme, verifying the effectiveness of the "residual hybrid gating" of this invention and laying a solid foundation for building a unified comprehensive health state detection framework.

[0054] Table 1. Accuracy of driver multi-task state detection for each model under different input modal combinations.

[0055]

Claims

1. A multimodal driver state detection method based on residual hybrid gating, characterized in that, The multimodal data undergoes synchronous preprocessing, local and global feature extraction, and residual hybrid gating fusion sequentially. After pooling and a feedforward classifier, the driver's comprehensive health status is output. Finally, a personalized comprehensive health navigation method for drivers is used to optimize the status. The specific implementation steps are as follows: Step 1: Data synchronization preprocessing; The sequences of the four primitive modes are as follows: ； in Represents the original data. , , and These represent the face, electrocardiogram, vehicle, and road, respectively. Represents the time step; preprocessing includes resampling alignment, noise reduction filtering, and normalization, which can be abstracted into the following formula: ； in, Indicates based on common sampling rate Perform resampling alignment. Indicates noise reduction filtering. Represents standardized processing; For the preprocessed data, To standardize sequence length, For the first The number of channels for each modality; the modalities are Face, ECG, Vehicle, and Road. Step 2: Local-Global Joint Feature Extraction; Local-global joint feature extraction: Within each modality, a lightweight CNN and a DyT-Transformer two-level encoder are introduced: the shallow CNN is responsible for local information detection, and the deep DyT-Transformer is responsible for global dependency modeling. Step 3: Residual hybrid gating fusion; Residual hybrid gating fusion: When a certain modality is missing or distorted, its weight is reduced; Fully utilize complementarity: When other modalities provide discriminative information, they are supplemented as needed; Maintain residual paths: Prevent over-fusion from diluting the features of the main modality; Step 4: Classify health status; ； in, Indicates the weight of the category header. Indicates bias. for The predicted probability of a health status; multimodal fusion features are mapped to interpretable probability outputs to achieve a comprehensive judgment of the driver's emotions, stress, fatigue, and distraction.

2. The multimodal driver state detection method based on residual hybrid gating according to claim 1, characterized in that, In step two, firstly, a batch normalization (BN) layer is cascaded after the convolutional layer to reduce gradient explosion / vanishing and stabilize the local feature extraction stage. Secondly, ReLU is used as the activation function to drive the convolutional channels to learn sparse activation patterns, highlighting sudden facial expressions and transient ECG fluctuations, thus improving the nonlinear expressive power of shallow networks. Finally, Dropout is used to suppress overfitting of local convolutional branches to a few significant channels and for lightweight regularization, without increasing inference overhead. The local convolutional encoding formula is abstractly represented as follows: ； in, This represents two layers of one-dimensional convolution, BN, ReLU, and Dropout. This indicates the learnable parameters of the convolutional branch; Represents local spatiotemporal details; While maintaining the same temporal length, the number of channels in the convolution output is mapped to a unified embedding dimension. This reduces subsequent computation and facilitates cross-modal alignment. The formula is as follows: ； in, yes Convolution weights, This indicates that the projected features have channels aligned to the shared dimension. ; Dynamic-Tanh normalization, compared to layer normalization, can reduce the computational complexity of division and square root operations, thereby reducing latency and memory usage without sacrificing accuracy; the DyT expression is as follows: ； in, Represents the input vector (the first one) Frame, channel dimension ), Indicates the learnable amplification factor. Indicates the tangent of the entire curve, with an output range of... , This represents the scaling and translation parameters per channel. This represents a normalized vector; A DyT-Transformer global encoder is introduced at the back end of each modality: long-range relationships are explicitly modeled through a self-attention mechanism to reveal high-order semantic dependencies between different time segments, as expressed below: ； in, This represents a learnable position vector, which injects positional information into the sequence; Indicates by A network composed of layers of DyT-Transformers, Indicates learnable parameters, It represents global dependency features, which imply long-term temporal sequence and contextual semantics.

3. The multimodal driver state detection method based on residual hybrid gating according to claim 1, characterized in that, In step three, cross-attention is first used to extract bidirectional cross-modal information, accurately align the temporal context, and explicitly model cross-modal dependencies; ； in, Indicates the dominant mode. Indicates auxiliary mode, Indicates "dominant mode" "Auxiliary modality" features cross-modal interaction; its shape remains consistent with the primary modality. Secondly, retention factors and update factors are generated through a gated weight generation network, as follows: ； ； in, Indicates global average pooling. This indicates a splicing operation between the primary and secondary modes. This represents the weights and biases of the gated weight generation network. represent Function, output , Indicates the retention factor. Indicates the update factor; Then, residual mixing and fusion are performed as follows: ； in, This indicates element-wise multiplication. If the principal mode signal-to-noise ratio is high, then... A high proportion of these modalities means they retain more of their own information. If the auxiliary modality is more reliable, then... It has a higher proportion and incorporates more cross-modal information; It is the result of fusing the main mode with a single auxiliary mode; Dominant mode With the other 3 auxiliary paths After fusion, the average is calculated along the path dimension. This averaging operation ensures that the contributions of each path are equal, avoiding the introduction of additional parameters due to relearning the weights. The final cross-modal representation is as follows: ； in, The total number of modalities is used to obtain the final fused features from the four channels. ; Finally, temporal average pooling and multimodal vector concatenation are performed on the fused features to obtain the final fused features, which can be used for later classification, as shown below: 。 4. A multimodal driver state detection system based on residual hybrid gating, as described in any one of claims 1-3, characterized in that, For four health conditions, the driver's physiological and psychological state is regulated in a closed-loop manner to ensure that the driver is within the driving range; as follows: ； in, The personalized comprehensive health index is divided into four levels within the value range of 0-10, where 8-10 indicates a good health status, 6-7 indicates a fair health status, 4-5 indicates a moderate health status, and 0-3 indicates a poor health status. , , and These represent the driver's emotional state, stress, fatigue, and distraction levels, respectively, with values ​​ranging from 0 to 10. 8-10 indicates an excellent health level, 6-7 indicates a good to moderate health level, 4-5 indicates a moderate health level, and 0-3 indicates a poor health level. , , and These represent personalized weights for emotion, stress, fatigue, and distraction, and satisfy the following conditions: ; This represents the adjustment amount for the health status level, and satisfies... ; The numerical range and real-time correction amount based on the personalized comprehensive health index Health status optimization measures are divided into four levels: Level 1 optimization, Level 2 optimization, Level 3 optimization, and Level 4 optimization.

Images