Driver fatigue detection method and system based on multi-modal feature fusion

Abstract

The invention relates to a driver fatigue detection method and system based on multi-modal feature fusion. The method comprises the following steps: collecting a face video signal and a physiological signal of a driver; performing feature extraction on the collected video signals and physiological signals; performing feature fusion on the extracted video signal features and physiological signal features by using a Hadamard product feature fusion method to generate a fusion feature vector; and inputting the fusion feature vector into a full connection layer classifier, and judging the fatigue state of the driver. According to the embodiment of the invention, the face video signal and the physiological signal of the driver are collected, the extracted multi-modal features are fused through the Hadamard product feature fusion method, the fused features are input into the full-connection layer classifier, training is carried out in a supervised learning mode, and the fatigue state of the driver is detected in real time. The accuracy and robustness of detection are improved, fatigue early warning can be carried out in an actual driving scene, and effective guarantee is provided for traffic safety.

Description

Technical Field

[0001] The disclosed embodiments relate to the technical field of fatigue detection, and in particular to a driver fatigue detection method and system based on multimodal feature fusion. Background Art

[0002] Driving fatigue is one of the main causes of traffic accidents, especially in the case of long-distance driving or night driving, the driver's mental and physical strength will gradually decline, leading to fatigue. Traditional driving fatigue detection methods are mostly based on a certain modal feature of the driver, such as facial expression, eye closure frequency, head posture, heart rate, breathing rate, etc. These detection methods have their own advantages and disadvantages. However, in actual driving environments, single-modal detection methods often have the following problems:

[0003] (1) Video information is easily affected by the environment: video-based fatigue detection methods (such as facial expression, eye closure detection, etc.) often fail when the light changes greatly or the camera angle is not good. For example, when driving at night or under strong light, the image collected by the camera may be of low quality, resulting in reduced detection accuracy. In addition, when the driver wears glasses, covers the face, or the facial posture changes greatly, the fatigue detection model based on facial features is difficult to provide stable judgment results.

[0004] (2) Singleness of physiological signals: Fatigue detection methods based on physiological signals (such as heart rate, galvanic skin response, brain waves, etc.) usually require additional sensing equipment, which will cause great interference to the driver after integration with the vehicle. At the same time, a single physiological signal such as heart rate will also fluctuate greatly during exercise, emotional fluctuations, etc., which is prone to false alarms and cannot reliably determine the fatigue state alone. In addition, the characteristics of physiological signals often change gradually over a long period of time, and their response often lags behind the driver's actual fatigue state and is easily affected by external factors. Detection methods that rely solely on physiological signals may not be sufficient in terms of real-time performance.

[0005] (3) Limitations of single-modality methods: Single-modality methods have limited applicability in complex scenarios and are easily affected by various interference factors. For example, detection methods based solely on video may ignore the driver's internal physiological changes, while detection methods based on physiological signals may not reflect the driver's external behavioral characteristics. Due to the complexity of fatigue status, any single-modality data source is difficult to accurately reflect the driver's overall status.

[0006] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0007] It should be noted that this section is intended to provide background or context for the technical solutions of the present disclosure stated in the claims. The description herein is not admitted to be prior art by virtue of being included in this section. Summary of the invention

[0008] The purpose of the embodiments of the present disclosure is to provide a driver fatigue detection method and system based on multimodal feature fusion, thereby overcoming one or more problems caused by the limitations and defects of related technologies at least to a certain extent.

[0009] The present disclosure first provides a driver fatigue detection method based on multimodal feature fusion, including:

[0010] Collect the driver's facial video signal and physiological signal;

[0011] Performing feature extraction on the collected video signal and physiological signal, wherein the feature extraction on the video signal is performed by a convolutional neural network, and the time series feature extraction on the physiological signal is performed by a transformer model;

[0012] The Hadamard product feature fusion method is used to fuse the extracted video signal features and physiological signal features to generate a fused feature vector;

[0013] The fused feature vector is input into a fully connected layer classifier, and the driver's fatigue state is judged by the fully connected layer classifier.

[0014] In one embodiment of the present disclosure, the video signal includes: driver's facial expression, eye closure frequency and head movement; the physiological signal includes: heart rate, electroencephalogram and breathing frequency.

[0015] In one embodiment of the present disclosure, the step of collecting the driver's facial video signal and physiological signal includes:

[0016] During the acquisition process, the time of the video signal and the physiological signal are synchronized using a timestamp.

[0017] In one embodiment of the present disclosure, the step of extracting features from the video signal using a convolutional neural network includes:

[0018] Use the first convolutional layer to perform a convolution operation on the input image, and use the first pooling layer to reduce the size of the feature map;

[0019] Using a second convolutional layer and a second pooling layer to extract high-level visual features in the image until spatial features in the video are generated;

[0020] Extracting global information from the image and performing feature dimension reduction processing to generate a global information feature vector;

[0021] Extract and summarize the motion features of the optical flow information in the video to generate a motion information feature vector;

[0022] The global information feature vector and the motion information feature vector are feature fused.

[0023] In one embodiment of the present disclosure, the step of extracting time series features of the physiological signal through a transformer model includes:

[0024] The spatial attention encoder is used to capture the multi-channel dependencies of the input physiological signals;

[0025] After transposing the physiological signals, the temporal attention encoder is used to capture the dependencies and important time point features in the temporal dimension.

[0026] The physiological signal is convolutionally processed using a convolutional neural network feature extraction module.

[0027] In one embodiment of the present disclosure, the step of fusing the extracted video signal features and physiological signal features using the Hadamard product feature fusion method to generate a fused feature vector includes:

[0028] The Hadamard product feature fusion method is used to multiply each vector in the extracted video signal features and physiological signal features element by element to generate a fused feature vector.

[0029] In one embodiment of the present disclosure, the fully connected layer classifier adopts a fully connected neural network classification model, and the process of training the classification model before using the classification model includes:

[0030] Collecting the video signal and the physiological signal to obtain a training data set;

[0031] Labeling each data sample in the training data set with a corresponding fatigue state label;

[0032] Process the data samples after annotating the fatigue state label.

[0033] In one embodiment of the present disclosure, the fatigue status labels include: awake, mild fatigue and severe fatigue.

[0034] In one embodiment of the present disclosure, the detection method further includes:

[0035] When the driver's fatigue status is mild fatigue or severe fatigue, an alarm is issued or safety measures are implemented.

[0036] The embodiment of the present disclosure also provides a driver fatigue detection system based on multimodal feature fusion, the detection system comprising:

[0037] The vehicle-mounted camera is used to collect the driver's facial video signal;

[0038] A physiological signal collector, used to collect the driver's physiological signals;

[0039] A feature extraction module, used to extract features from the collected video signals and physiological signals, wherein the video signals are subjected to feature extraction through a convolutional neural network, and the physiological signals are subjected to temporal feature extraction through a transformer model;

[0040] A feature fusion module is used to fuse the extracted video signal features and physiological signal features using a Hadamard product feature fusion method to generate a fused feature vector;

[0041] The fully connected layer classifier is used to receive the fused feature vector and determine the driver's fatigue state.

[0042] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0043] The driver fatigue detection method and system based on multimodal feature fusion in the disclosed embodiment collects the driver's facial video signal and physiological signal, then uses a convolutional neural network to extract features from the video signal, extracts temporal features from the physiological signal through a transformer model, and fuses the extracted multimodal features through a Hadamard product feature fusion method. The fused features are input into a fully connected layer classifier, and are trained in a supervised learning manner to detect the driver's fatigue state in real time. This application improves the accuracy and robustness of detection through multimodal fusion, and can provide fatigue warnings in actual driving scenarios, providing effective protection for traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0045] Figure 1 A schematic flow chart of a driver fatigue detection method based on multimodal feature fusion in this embodiment is shown;

[0046] Figure 2 A schematic diagram of a process of extracting features from the video signal using a convolutional neural network in this embodiment is shown;

[0047] Figure 3A schematic diagram of a process of extracting time series features of the physiological signal through a transformer model in this embodiment is shown;

[0048] Figure 4 A schematic diagram showing a process of training a fully connected neural network classification model before using the fully connected layer classifier in this embodiment is shown;

[0049] Figure 5 The following is a framework diagram of a multimodal feature fusion model based on deep learning in this embodiment;

[0050] Figure 6 A Two-stream CNN (two-stream convolutional neural network) model diagram in this embodiment is shown;

[0051] Figure 7 A schematic diagram of the physiological signal feature extraction process in this embodiment is shown;

[0052] Figure 8 The structure and workflow of the multi-head spatial attention encoder in this embodiment are shown;

[0053] Fig. 9 The structure and workflow of the single-head spatial attention mechanism in this embodiment are shown;

[0054] Fig.10 The structure and workflow of the multi-head slicing mechanism in the time dimension in this embodiment are shown;

[0055] Fig.11 The structure and workflow of the multi-head spatial attention mechanism in this embodiment are shown;

[0056] Fig.12 The structure and workflow of the multi-head temporal attention encoder in this embodiment are shown;

[0057] Fig.13 The structure and workflow of the single-head temporal attention mechanism in this embodiment are shown;

[0058] Fig.14 The structure and workflow of the multi-head slicing mechanism in the channel dimension in this embodiment are shown;

[0059] Fig.15 The structure and workflow of the multi-head temporal attention mechanism in this embodiment are shown;

[0060] Fig.16 The structure and workflow of the convolutional neural network feature extraction module in this embodiment are shown;

[0061] Fig.17 A schematic diagram of the structure of an electronic device in this embodiment is shown;

[0062] Fig.18 A schematic diagram showing the structure of a program product for implementing a driver fatigue detection method based on multimodal feature fusion in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0063] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0064] In addition, the accompanying drawings are only schematic illustrations of the embodiments of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0065] In this example implementation, a driver fatigue detection method based on multimodal feature fusion is first provided. Figure 1 , may include: Step S101-Step S104, which are specifically as follows:

[0066] Step S101, collecting facial video signals and physiological signals of the driver. The video signals include: driver's facial expressions, eye closure frequency, head movements, etc.; the physiological signals include: heart rate, electroencephalogram, and breathing rate, etc., such as electroencephalogram (EEG) data, electromyogram (EMG) data, electrooculogram (EOG) data, and electrocardiogram (ECG) data.

[0067] Step S102, extracting features from the collected video signal and physiological signal, wherein the video signal is feature extracted by a convolutional neural network, and the physiological signal is time-series feature extracted by a transformer model. Specifically, a Two-stream CNN can be used to extract features from the collected video signal and physiological signal.

[0068] Step S103, the extracted video signal features and physiological signal features are fused using the Hadamard product feature fusion method to generate a fused feature vector. The Hadamard product is an element-by-element multiplication operation. Specifically, for the feature vectors obtained from the video signal and the physiological signal, the Hadamard product method multiplies the corresponding elements in each vector to generate a new fused feature vector. Through this element-by-element multiplication operation, the information in the video and physiological signals can be effectively combined, and the correlation between the two modalities can be retained in the fused features.

[0069] Step S104, input the fused feature vector into a fully connected layer classifier, and use the fully connected layer classifier to determine the driver's fatigue state, wherein the driver's fatigue state includes: awake, mild fatigue and severe fatigue.

[0070] In this embodiment, the facial video signal and physiological signal of the driver are collected, and then the video signal is feature extracted using a convolutional neural network, the physiological signal is time-series feature extracted using a transformer model, and the extracted multimodal features are fused using the Hadamard product feature fusion method. The fused features are input into the fully connected layer classifier, and the driver's fatigue state is detected in real time by training in a supervised learning manner. This application improves the accuracy and robustness of detection through multimodal fusion, and can provide fatigue warning in actual driving scenarios, providing effective protection for traffic safety.

[0071] Combine the following Figure 1-Figure 16 Each step of the present application is described in more detail.

[0072] In step S101, during the process of driving fatigue detection, the driver's multimodal data needs to be collected first, including video signals and multiple physiological signal data. Video data is collected through the in-car camera to capture the driver's facial expressions, eye closure frequency, head posture and other behavioral characteristics. Physiological signals are obtained through physiological sensors, mainly including electroencephalogram (EEG), electromyogram (EMG), electrooculogram (EOG) and electrocardiogram (ECG). EEG is used to monitor the activity state of the brain, which can reflect the driver's attention and fatigue state; EMG monitors the electrical activity of muscles, especially the muscle reaction of the face and neck; EOG is used to capture the movement of the eyeballs and can identify the driver's blinking frequency and eye movement characteristics; ECG is used to monitor heart rate and heart rate variability to detect the driver's tension or relaxation state. In order to ensure the synchronization of multimodal data, video signals and physiological signals need to be time-aligned during the acquisition process, that is, the data of different signal sources are synchronized through timestamp technology, so that the video frames and physiological signals in each time period correspond to the driver's state at the same time. This time alignment method provides a basis for subsequent feature extraction and fusion, ensuring the spatiotemporal consistency of multimodal data.

[0073] In step S102, feature extraction is performed on the collected video signal and physiological signal respectively.

[0074] 1. Video Feature Extraction

[0075] The video feature extraction uses a two-stream CNN model, which is mainly divided into two parallel branches: one branch processes image frames (static frames), and the other branch processes the optical flow information of the video (motion frames). The extraction of these two features can capture the temporal and spatial information in the video at the same time.

[0076] Please refer to Figure 2 , in the image frame branch, the feature extraction steps include S201 to S203:

[0077] S201, using a first convolutional layer to perform a convolution operation on the input image, and using a first pooling layer to reduce the size of the feature map. The first convolutional layer may be a 7x7 convolutional layer, and the first pooling layer may be a pooling layer with a step size of 2.

[0078] S202, extracting high-level visual features in the image using a second convolutional layer and a second pooling layer until generating spatial features in the video, wherein the second convolutional layer and the second pooling layer process deeper convolution and pooling operations than the first convolutional layer and the first pooling layer.

[0079] S203, extracting the global information in the image and performing feature dimension reduction processing to generate a global information feature vector.

[0080] Specifically, the input image is first convolved through a 7x7 convolution layer (Conv1), and a pooling layer (Max pooling) with a stride of 2 is used to reduce the size of the feature map and retain the main features. Next, through a series of deeper convolution layers (such as Conv2, Conv3, Conv4, etc.) and pooling operations, high-level visual features in the image are gradually extracted. These convolution layers enable the model to capture local features of the driver's face, such as eye closure, head posture, etc., by continuously increasing the convolution kernel size and feature map depth, and gradually generate spatial features in the video. In the last two fully connected layers (FC6 and FC7), the model further extracts global information from the image and performs feature dimensionality reduction. In order to enhance the generalization ability of the model, a dropout layer is used to prevent overfitting.

[0081] Another branch, the step of extracting motion features by processing optical flow information in the video includes: S204 and S205.

[0082] S204, extracting and summarizing motion features of the optical flow information in the video to generate a motion information feature vector.

[0083] S205: Perform feature fusion on the global information feature vector and the motion information feature vector.

[0084] Specifically, optical flow describes the motion changes of pixels in an image over time, so it can capture dynamic features of the driver's face, such as eye movement, head rotation, etc. In this branch, the optical flow input also undergoes multi-layer convolution and pooling operations similar to the image frame branch to extract motion features that change over time. The convolution layer design of this branch is consistent with the image frame branch, including the first layer (Conv1) of the 7x7 convolution kernel, followed by deeper convolution layers and pooling layers to gradually extract and filter motion information in the video. Finally, the motion features are aggregated through the fully connected layer to generate a feature vector representing the motion information in the video, with a maximum vector length of 2048.

[0085] After the two branches have been extracted from their own features, they will perform feature fusion (Concat) after the fully connected layer. By splicing the feature vectors of the RGB frame and the optical flow frame together, the model can simultaneously capture the static spatial information and dynamic temporal information in the video, thereby providing a high-dimensional feature representation containing spatiotemporal features for the subsequent classifier.

[0086] For the calculation of optical flow, this application recommends the use of the Farneback algorithm, which is a classic dense optical flow calculation method that can calculate the motion vector for each pixel in the video. The Farneback algorithm estimates the local motion of the image at different scales by constructing a Gaussian pyramid, and calculates the accurate optical flow vector through multi-level refinement. This method is fast and stable, and is suitable for real-time video processing scenarios.

[0087] The core of the Farneback algorithm is to estimate motion by the change in pixel intensity between two consecutive image frames. Assuming there are two consecutive image frames I(t) and I(t+1), the Farneback algorithm uses a local image window to fit a quadratic polynomial to approximate the local image intensity change:

[0088] I(x)≈x T Ax+b T x+c

[0089] Among them, x is the coordinate (x, y) in the image; A is the symmetric gradient matrix, which represents the second-order partial derivative of the image; b is the first-order gradient vector of the image; c is a constant term. By comparing the difference between the two frames of images, the Farneback algorithm estimates the motion vector (i.e., optical flow) of each pixel:

[0090] v=A -1 b

[0091] Among them, v is the motion vector, which indicates the direction and size of the optical flow of the current pixel; A -1 is the inverse matrix of the image gradient matrix, which is used to calculate the local transformation; b is the gradient information of the current image block.

[0092] 2. Physiological Signal Feature Extraction

[0093] (1) Overall framework of physiological signal processing

[0094] Please refer to Figure 7 In the physiological signal feature extraction process of the present invention, a spatial attention encoder, a temporal attention encoder and a convolutional neural network feature extraction block (CNN-Feature Extraction Block) are utilized. Through a multi-level feature extraction method, the spatiotemporal features in the physiological signals are comprehensively analyzed, thereby improving the accuracy of fatigue detection.

[0095] The input physiological signals usually have multi-channel time series features, such as EEG, ECG and other signals, and their shape is [B,C,T], where B is the batch size, C is the number of channels, and T is the time step. In order to effectively extract the spatiotemporal features in these signals, the steps of extracting time series features from the physiological signals through the transformer model are described in detail. Figure 3 , including the following steps:

[0096] S301, using a spatial attention encoder to capture multi-channel dependencies of input physiological signals to highlight spatial features.

[0097] S302, after transposing the physiological signal, a temporal attention encoder is used to capture its dependencies in the time dimension and important time point features.

[0098] S303, performing convolution processing on the physiological signal using a convolutional neural network feature extraction module to further extract local spatiotemporal features through a convolutional layer.

[0099] (2) Multi-head spatial attention encoding

[0100] Figure 8 The structure and workflow of the spatial attention encoder are presented to extract features of the spatial dimension (channel dimension) from the input physiological signals. The main goal of the spatial attention encoder is to capture the dependencies and importance between different channels in the physiological signals. Especially for multi-channel signals such as EEG, the information coordination between channels is crucial to the accuracy of the task. Figure 8 Each part is described in detail.

[0101] First, the shape of the input data is [B, C, T], where B represents the batch size, that is, the number of samples processed at a time; C represents the number of channels of the signal, which usually corresponds to different measurement channels of physiological signals (such as different electrode positions of EEG signals); T represents the time step, that is, the length of the time series data on each channel. The input data contains information about the signal in the time and channel dimensions. The main task of the encoder is to extract spatial features by focusing on the dependencies between channels.

[0102] Next, the input data passes through the Multi-head Spatial Attention mechanism, which allows the model to calculate the attention weights between channels in parallel in different subspaces. Each attention head calculates the correlation of the input signal on different channels, thereby assigning different importance weights to the channels. Through the multi-head attention mechanism, the model can focus on those channels that are most relevant to the task and ignore or weaken those channels that are unimportant or noisy. After the calculation is completed, a weighted feature matrix is ​​generated, which has the same shape as the original input data, but the features on the channels have been weighted and adjusted.

[0103] To ensure the stability of the model and prevent the gradient from disappearing or exploding, the encoder uses residual connections and normalization layers (Add&Norm). The residual connection ensures the continuous transmission of information by adding the output of the attention mechanism to the original input data, while the normalization layer standardizes the results, making the model training more efficient and stable.

[0104] The core idea of ​​the multi-head spatial attention mechanism is to extract spatial features from different subspaces by computing multiple attention heads in parallel, so as to enhance the model's ability to capture information from different channels. In physiological signal analysis, signals from different channels (such as different electrodes of EEG signals) may contain complex correlation information. Therefore, the multi-head attention mechanism can effectively filter and weight the information of each channel and enhance the expression ability of important channels.

[0105] First, let’s introduce the calculation process of single-head attention. Fig. 9 In , a single-head spatial attention operates on the inputs Q (query matrix), K (key matrix), and V (value matrix). The attention head calculates the similarity between the query matrix Q and the key matrix K to form an attention score, which is then multiplied by the value matrix V to output the weighted result. To improve numerical stability, the similarity calculation result is divided by Then, the attention distribution of each channel is generated through softmax normalization. The process can be expressed as:

[0106]

[0107] Next, we introduce the calculation process of single-head spatial attention. First, the input data is mapped into Q, K, and V matrices respectively through the linear transformation layer. Then the Q, K, and V matrices are split along the time T dimension into h heads, and each attention head is calculated independently. Fig.10The details of the multi-head slicing mechanism are shown. Each attention head slices the input Q, K, V to form multiple sub-matrices, each representing a different time segment. Through this slicing mechanism, the time series data can be decomposed into multiple time periods so that the model can perform attention calculations in a finer-grained time range. In multi-head attention, each head corresponds to a slice (such as Q1 / K1 / V1, Q2 / K2 / V2, until Q h / K h / V h ), these slices calculate the spatial attention score of each channel respectively. The output results of multiple attention heads are merged at the end to form a complete spatiotemporal attention feature.

[0108] at last, Fig.11 The output structure of the multi-head attention mechanism is shown. After the attention calculation of each time segment is completed, the model will merge the results of all attention heads to form a complete attention feature matrix. By combining different attention heads, the model can capture multiple spatial subspace dimensions of the input signal, enhancing the signal's ability to express spatiotemporal features.

[0109] The merged attention features still maintain the same shape as the input data [B, C, T], but the information of each channel has been weighted and adjusted, important channels are highlighted, and the influence of noise channels is suppressed. This design enables the multi-head attention mechanism to better process complex physiological signals, especially the spatial relationship of multi-channel physiological signals, such as the dependency between different electrodes of EEG signals.

[0110] like Figure 8 After the multi-head attention mechanism and residual connection, the data is passed to a two-layer feedforward neural network, which further extracts high-order features from the input signal. The feedforward neural network contains two layers of linear transformation layers and activation function layers, which enhance the expressiveness of the model through nonlinear operations and capture more complex inter-channel dependencies.

[0111] The data processed by the feedforward neural network is again subjected to residual connection and normalization to ensure that the output of the model still retains the key information in the input signal while improving the effect of feature expression. The final output shape remains the same as the input [B, C, T], but after multi-head attention and feedforward neural network processing, the channel features have been enhanced, and the dependencies between channels have been fully captured and utilized.

[0112] In general, the spatial attention encoder, through the combination of multi-head attention mechanism, feedforward neural network and residual connection, can effectively identify important channel features in physiological signals and highlight those signals that are closely related to fatigue detection tasks, thereby improving the detection accuracy and robustness of the entire model.

[0113] (3) Multi-head spatial attention encoding

[0114] After completing the spatial attention mechanism, the physiological signal is temporally transposed to process the features in the time dimension. After that, the signal enters the Temporal Attention Encoder, which is used to capture the changes in the signal in the time dimension. Temporal physiological signals (such as EEG, ECG) usually contain a large number of time-related pattern changes, such as the rhythm of brain waves, fluctuations in heart rate, etc. The temporal attention mechanism can automatically assign weights to different time steps and highlight those important time segments related to fatigue. This step can significantly improve the analysis ability of complex time series signals by modeling the features of physiological signals at different time points.

[0115] like Fig.12 As shown in the figure, the structure and workflow of the Multi-head Temporal Attention Encoder are as follows: First, the input signal passes through the position encoding, which provides the input data with information about the time step so that the model can recognize the position and order of different time points. This is crucial for processing time series data because the position encoding can help the model perceive the sequential relationship in the time series.

[0116] Next, the input data enters the Multi-head Temporal Attention mechanism, which is the core part of the encoder. It processes the input time series data by calculating multiple attention heads in parallel. Each attention head calculates the dot product between the query matrix Q, the key matrix K, and the value matrix V to generate the attention weights in time. By normalizing the weights through the softmax function, the model can identify which moments in the time series are most important for the current task. Subsequently, the attention weights are multiplied by the value matrix V to generate weighted time features. These features capture the dependencies and change patterns of the input signal in the time dimension.

[0117] After multi-head attention calculation, the features are processed through residual connection (Add) and normalization (Norm) to ensure that the original input information is preserved and prevent information loss in the deep network. The residual connection adds the output of the multi-head attention to the original input data, and the normalization standardizes the result to ensure the stability of the training process.

[0118] Next, the processed data is further extracted through a feedforward neural network (FAN). The FAN consists of two layers of linear transformation and nonlinear activation function, which can perform more complex transformations on input features and enhance the expressiveness of the model. Finally, the output of the FAN is again processed through residual connection and normalization to ensure the training stability and feature continuity of the entire model.

[0119] The structure of the entire multi-head temporal attention encoder can extract key time-dependent features from time series data through the combination of parallel processing of multiple attention heads, residual connections and feedforward networks. These features can provide strong support for subsequent classification tasks and are particularly suitable for tasks that need to capture temporal changes, such as detecting driver fatigue.

[0120] The process of temporal multi-head attention encoding is similar to that of spatial multi-head attention, except that the input data is transposed. Fig.13 In , a single-head temporal attention operates on the inputs Q (query matrix), K (key matrix), and V (value matrix). The attention head calculates the similarity between the query matrix Q and the key matrix K to form an attention score, which is then multiplied by the value matrix V to output the weighted result. To improve numerical stability, the similarity calculation result is divided by Then, the attention distribution of each channel is generated through softmax normalization. The process can be expressed as:

[0121]

[0122] like Fig.14 As shown in Figure 2, the multi-head C-Slice module in the multi-head temporal attention mechanism slices the input Channel dimension data. Each slice Q h ,K h ,V h Represents an attention head, and each attention head can focus on capturing different temporal patterns. Through this sharding mechanism, the model can process temporal information in multiple subspaces in parallel, thereby extracting different features in different time segments.

[0123] Afterwards, if Fig.15As shown in Figure 1, after the attention calculation of each head, these attention scores are multiplied by the value matrix V to generate weighted time features. These features are combined at the end, and the outputs of different attention heads are merged through a linear layer to generate the final output feature matrix. The merged output data still maintains the same shape as the input [B, C, T], but the features in the time dimension have been re-weighted, enhancing the information of important time steps and weakening noise or unimportant time points.

[0124] One advantage of the multi-head temporal attention mechanism is that it allows the model to learn complex temporal dependency patterns in parallel on different temporal subspaces. Through the parallel computation of multiple attention heads, the model can flexibly capture the dynamic changes between different time steps in physiological signals and associate these changes with tasks such as fatigue status. Ultimately, these weighted temporal features provide more discriminative input features for subsequent classification or decision making.

[0125] (4) Convolutional neural network feature extraction module

[0126] The features processed by the spatiotemporal attention mechanism are transmitted to the convolutional neural network feature extraction module (CNN-Feature Extraction Block). The CNN module extracts local spatial and temporal high-order features from the signal through a series of convolution operations. The convolution layer can automatically extract discriminative patterns in physiological signals, such as frequency components and waveform changes in EEG signals. These high-order features are further integrated as input to subsequent classifiers or fatigue detection models, providing an accurate basis for the final fatigue state classification.

[0127] Fig.16 The structure and workflow of the convolutional neural network feature extraction module (CNN-Feature Extraction Block) are shown. This module is used to extract temporal features from input physiological signals. Its core is to gradually extract and compress the feature representation of the data through one-dimensional convolution (1DConv) and maximum pooling (Max Pooling) operations, and finally output the high-dimensional features extracted by the convolution layer.

[0128] After passing through the spatiotemporal attention encoder, the shape of the input data is [B, C, T]. These input signals contain rich temporal information. The convolutional neural network module gradually extracts the temporal dependency features in the signal through multiple convolution and pooling layers.

[0129] The data is split in the C dimension. On each branch, feature extraction is first performed through a one-dimensional convolution layer with kernel sizes of 7 and 5 and a step size of 2. These convolution layers are used to capture local patterns in the signal and extract different features of the signal through multiple convolution kernels, such as frequency components in the time series, waveform changes, etc. The convolution operation also effectively reduces the number of model parameters and improves computational efficiency through the weight sharing mechanism. After each convolution layer, a maximum pooling operation is used with a pooling window of 1×21\times 21×2. Maximum pooling is used to downsample the features after convolution, retain the main features, and reduce the dimension of the feature map, reducing the amount of calculation and the risk of overfitting. Through maximum pooling, the key information in the signal is further compressed and redundant information is discarded. This process is repeated multiple times in each branch. After each layer of convolution and pooling operations, the time dimension T of the feature map is gradually reduced and the feature dimension increases.

[0130] After all the convolution and pooling layers, the outputs of the three branches are flattened into one-dimensional vectors through the Flatten operation. Then, these flattened features are concatenated in the Concat layer to combine the features extracted by different branches to form a high-dimensional feature vector.

[0131] The concatenated features are input into a fully connected layer (FC layer) with 2048 neurons for further nonlinear transformation of the combined features. The function of the fully connected layer is to integrate the local features extracted by the convolutional layer and provide high-level feature representation for subsequent classification tasks.

[0132] Through multiple layers of convolution, pooling, flattening and concatenation, the convolutional neural network feature extraction module can extract discriminative features from temporal physiological signals. Finally, these features are further processed through the fully connected layer to provide high-quality input data for tasks such as fatigue detection.

[0133] The process of feature fusion of features extracted from video signals and physiological signals is described in detail below.

[0134] After the features of the video and physiological signals are extracted separately, the Hadamard product feature fusion method is used to fuse the features of the two modalities. Specifically, the features of the video signal are usually extracted through a convolutional neural network (CNN) or a two-stream convolutional neural network (two-stream CNN) network to capture the driver's facial expressions, head movements, eye movements and other visual features; the features of the physiological signal are extracted through an encoder based on a transformer or convolutional network, which contains physiological change information such as heart rate, brain waves, and breathing rate. The features of these two modalities each reflect the changes in the driver's external behavior and internal physiological state.

[0135] The advantage of the Hadamard product fusion method is that it can highlight the important features in both the video and the physiological signal. If the features of both modalities have high response values ​​at certain positions, the fused features will be enhanced; conversely, if a modality has no important information at a specific position, the feature value at that position will be weakened. This element-by-element multiplication method can fully capture the complementarity and synergy between the two modalities, helping the model to more accurately analyze the driver's fatigue state.

[0136] Through Hadamard product feature fusion, the fused feature vector contains the key information of video and physiological signals, providing high-quality input data for subsequent classifiers or decision models, and improving the accuracy and robustness of fatigue detection.

[0137] The construction process of the next fully connected layer classifier is as follows.

[0138] After the features of the video signal and the physiological signal are fused through the Hadamard product, the resulting fused features will be sent as input to the fully connected layer classifier to detect the driver's fatigue state. The task of the fully connected layer classifier is to determine whether the driver is currently in a fatigued state or awake state based on the fused features.

[0139] In order to train the fully connected layer classifier model, the fatigue status of the subjects needs to be pre-labeled during the data collection stage. Specifically, a standard fatigue scale is used to evaluate the status of each subject, and their fatigue status is divided into three categories: awake, mild fatigue, and severe fatigue. In the actual data collection process, video signal data (such as facial expressions, eye closure frequency, head posture, etc.) and physiological signal data (such as heart rate, EEG, breathing rate, etc.) are collected synchronously, and each data sample is labeled according to the evaluation results of the fatigue scale.

[0140] With these labeled training data, the fully connected layer classifier can be trained through supervised learning. The model will learn the mapping relationship between video signal and physiological signal features and fatigue status labels, and optimize the ability to judge fatigue status by adjusting model parameters. After training, the classifier can accurately predict the driver's current fatigue status through real-time input fusion features during actual driving.

[0141] In this way, the system can perform accurate driving fatigue detection based on the input of multimodal data (video signals and physiological signals) and Hadamard product fusion features using a fully connected layer classifier to help prevent safety risks caused by driving fatigue.

[0142] The model training and deployment process is described in detail below.

[0143] The fully connected layer classifier adopts a fully connected neural network classification model, please refer to Figure 4 The process of training the classification model before using the classification model includes the following steps:

[0144] Step S401, collecting the video signal and the physiological signal to obtain a training data set;

[0145] Step S402, labeling each data sample in the training data set with a corresponding fatigue state label;

[0146] Step S403: Process the data samples labeled with the fatigue status label.

[0147] Specifically, by collecting a large amount of multimodal data (including video signals and physiological signals) to train the model, a deep learning model that can predict the driver's fatigue state can be obtained. During the training phase, the system will collect a large amount of labeled sample data. Each sample will be labeled according to the fatigue scale and classified as awake, mildly fatigued, or severely fatigued. By using these labeled data, the model can learn the relationship between the video signal features and the physiological signal features, and finally train a model that can accurately predict the fatigue state.

[0148] Once the model is trained, it can be integrated into the actual driving environment to monitor the driver's status in real time through the on-board camera and physiological signal acquisition equipment. In actual applications, the on-board system will continuously collect the driver's facial video signal and physiological signal, and extract the corresponding features through the preprocessing step. The extracted multimodal features are fused through the Hadamard product feature, and the fused feature vector is input into the fully connected layer classifier in real time. The fully connected layer classifier determines the driver's current fatigue state through the fused features.

[0149] Based on the above embodiments, the system can monitor in real time whether the driver is awake, slightly fatigued, or severely fatigued based on the output of the fully connected layer classifier. Once the driver is detected to be in a fatigued state, the system can issue an alarm to remind the driver to rest, or even take further safety measures in the case of severe fatigue, such as automatically slowing down or controlling the vehicle. By combining the fatigue detection model with the on-board camera and physiological signal acquisition equipment, the system can effectively prevent traffic accidents caused by driver fatigue and improve driving safety.

[0150] An embodiment of the present application also provides a driver fatigue detection system, which includes: a vehicle-mounted camera, a physiological signal collector, a feature extraction module, a feature fusion module and a fully connected layer classifier.

[0151] Specifically, the on-board camera is used to collect the driver's facial video signal; the physiological signal collector is used to collect the driver's physiological signals; the feature extraction module is used to extract features from the collected video signals and physiological signals, wherein the video signals are feature extracted by a convolutional neural network, and the physiological signals are time series feature extracted by a transformer model; the feature fusion module is used to fuse the extracted video signal features and physiological signal features using the Hadamard product feature fusion method to generate a fused feature vector; the fully connected layer classifier is used to receive the fused feature vector and determine the driver's fatigue state.

[0152] In this embodiment, the facial video signal and physiological signal of the driver are collected, and then the video signal is feature extracted using a convolutional neural network, the physiological signal is time-series feature extracted using a transformer model, and the extracted multimodal features are fused using the Hadamard product feature fusion method. The fused features are input into the fully connected layer classifier, and the driver's fatigue state is detected in real time by training in a supervised learning manner. This application improves the accuracy and robustness of detection through multimodal fusion, and can provide fatigue warning in actual driving scenarios, providing effective protection for traffic safety.

[0153] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0154] It should be noted that although several modules of the system for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of a module described above can be further divided into multiple modules for concretization. The components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Those of ordinary skill in the art can understand and implement it without paying creative work.

[0155] See also Fig.17 The embodiment of the present invention further provides an electronic device 300, which includes at least one memory 310, at least one processor 320, and a bus 330 for connecting different platform systems.

[0156] The memory 310 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 311 and / or a cache memory 312 , and may further include a read-only memory (ROM) 313 .

[0157] Among them, the memory 310 also stores a computer program, which can be executed by the processor 320, so that the processor 320 executes the steps of the driver fatigue detection method based on multimodal feature fusion in any embodiment of the present invention. The specific implementation method is consistent with the implementation method and the technical effect recorded in the above-mentioned embodiment of the driver fatigue detection method based on multimodal feature fusion, and some contents will not be repeated here.

[0158] The memory 310 may also include a utility 314 having at least one program module 315, such program module 315 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0159] Accordingly, the processor 320 may execute the above-mentioned computer program and may execute the utility 314 .

[0160] Bus 330 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures.

[0161] The electronic device 300 may also communicate with one or more external devices 340 such as keyboards, pointing devices, Bluetooth devices, etc., and may also communicate with one or more devices that can interact with the electronic device 300, and / or communicate with any device (such as a router, a modem, etc.) that enables the electronic device 300 to communicate with one or more other computing devices. Such communication may be performed through an input / output interface 350. In addition, the electronic device 300 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 360. The network adapter 360 may communicate with other modules of the electronic device 300 through a bus 330. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0162] An embodiment of the present invention also provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed, it implements the steps of the driver fatigue detection method based on multimodal feature fusion in the embodiment of the present invention. The specific implementation method is consistent with the implementation method and the technical effect achieved in the above-mentioned embodiment of the driver fatigue detection method based on multimodal feature fusion, and some contents will not be repeated here.

[0163] Fig.18 The program product 400 provided in this embodiment for implementing the above-mentioned driver fatigue detection method based on multimodal feature fusion is shown, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product 400 of the present invention is not limited to this. In the present invention, the readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. The program product 400 can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0164] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or for use in combination with it. The program code contained on the readable storage medium may be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. The program code for performing the operation of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as C language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device, partially on a remote computing device, or entirely on a remote computing device or server. Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0165] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A driver fatigue detection method based on multimodal feature fusion, characterized in that: include: Collect the driver's facial video signal and physiological signal; Performing feature extraction on the collected video signal and physiological signal, wherein the feature extraction on the video signal is performed by a convolutional neural network, and the temporal feature extraction on the physiological signal is performed by a transformer model; The Hadamard product feature fusion method is used to fuse the extracted video signal features and physiological signal features to generate a fused feature vector; The fused feature vector is input into a fully connected layer classifier, and the driver's fatigue state is judged by the fully connected layer classifier.

2. The driver fatigue detection method based on multimodal feature fusion according to claim 1 is characterized in that: The video signal includes: driver's facial expression, eye closure frequency and head movement; the physiological signal includes: heart rate, electroencephalogram and breathing frequency.

3. The driver fatigue detection method based on multimodal feature fusion according to claim 1 is characterized in that: The step of collecting the driver's facial video signal and physiological signal comprises: During the acquisition process, the time of the video signal and the physiological signal are synchronized using a timestamp.

4. The driver fatigue detection method based on multimodal feature fusion according to claim 1 is characterized in that: The step of extracting features from the video signal using a convolutional neural network comprises: Use the first convolutional layer to perform a convolution operation on the input image, and use the first pooling layer to reduce the size of the feature map; Using a second convolutional layer and a second pooling layer to extract high-level visual features in the image until spatial features in the video are generated; Extracting global information from the image and performing feature dimension reduction processing to generate a global information feature vector; Extract and summarize the motion features of the optical flow information in the video to generate a motion information feature vector; The global information feature vector and the motion information feature vector are feature fused.

5. The driver fatigue detection method based on multimodal feature fusion according to claim 1 is characterized in that: The step of extracting time series features of the physiological signal through the transformer model includes: The spatial attention encoder is used to capture the multi-channel dependencies of the input physiological signals; After transposing the physiological signals, the temporal attention encoder is used to capture the dependencies and important time point features in the temporal dimension. The physiological signal is convolutionally processed using a convolutional neural network feature extraction module.

6. The driver fatigue detection method based on multimodal feature fusion according to claim 1 is characterized in that: The Hadamard product feature fusion method is used to fuse the extracted video signal features and physiological signal features. The steps of generating a fused feature vector include: The Hadamard product feature fusion method is used to multiply each vector in the extracted video signal features and physiological signal features element by element to generate a fused feature vector.

7. The driver fatigue detection method based on multimodal feature fusion according to claim 1 is characterized in that: The fully connected layer classifier adopts a fully connected neural network classification model. The process of training the classification model before using the classification model includes: Collecting the video signal and the physiological signal to obtain a training data set; Labeling each data sample in the training data set with a corresponding fatigue state label; Process the data samples after annotating the fatigue state label.

8. The driver fatigue detection method based on multimodal feature fusion according to claim 7 is characterized in that: The fatigue status labels include: awake, mild fatigue and severe fatigue.

9. The driver fatigue detection method based on multimodal feature fusion according to any one of claims 1 to 8, characterized in that: The detection method further comprises: When the driver's fatigue status is mild fatigue or severe fatigue, an alarm is issued or safety measures are implemented.

10. The driver fatigue detection system based on multimodal feature fusion is characterized by: The detection system comprises: The vehicle-mounted camera is used to collect the driver's facial video signal; A physiological signal collector, used to collect the driver's physiological signals; A feature extraction module, used to extract features from the collected video signals and physiological signals, wherein the video signals are subjected to feature extraction through a convolutional neural network, and the physiological signals are subjected to temporal feature extraction through a transformer model; A feature fusion module is used to fuse the extracted video signal features and physiological signal features using a Hadamard product feature fusion method to generate a fused feature vector; The fully connected layer classifier is used to receive the fused feature vector and determine the driver's fatigue state.

Images