Driver fatigue detection system based on multimodal Transformer network

Through the multimodal Transformer network, the driver's EEG signals and facial expressions are embedded and self-attention analysis is achieved, real-time detection of driver fatigue status is solved, and the problem of limited model performance in the existing technology is improved. Parallel performance and long-distance dependency modeling capabilities.

CN114782933BActive Publication Date: 2024-11-08ARTIFICIAL INTELLIGENCE RES INST OF HEFEI COMPREHENSIVE NAT SCI CENT (ANHUI ARTIFICIAL INTELLIGENCE LAB)
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202210501128.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-09
Publication Date
2024-11-08
Estimated Expiration
2042-05-09

AI Technical Summary

Technical Problem

The existing driver fatigue detection model is mainly single-modal, based on EEG or facial expression recognition, and there is room for improvement in the long-distance dependence and parallel performance of input features, resulting in Real-time detection system performance is limited.

Method used

The driver fatigue detection system based on the multimodal Transformer network is adopted to obtain EEG signals and facial expression images through the data acquisition module, and the embedded processing module is used for embedded processing. The self-attention analysis module uses the Transformer encoder to perform multi-head self-attention analysis. The result prediction module predicts the result through the prediction function and optimizes the prediction results through the loss function.

Benefits of technology

Real-time detection of driver fatigue state is realized, the parallel performance and long-distance dependency modeling capabilities of the model are improved, and the problem of inferring time-restricting model performance in the existing technology is solved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114782933B_ABST
    Figure CN114782933B_ABST
Patent Text Reader

Abstract

The present invention discloses a driver fatigue detection system based on a multimodal Transformer network, which involves deep learning and intelligent biomedical signal processing. A data acquisition module is used to obtain the driver's EEG signal and its facial expression image block; an embedding processing module is used to embed the EEG signal and the facial expression image block to obtain EEG tokens and image block tokens; a self-attention analysis module is used to perform multi-head self-attention analysis on the EEG tokens and image block tokens through a Transformer encoder to obtain EEG signal token vectors and image block token vectors; a result prediction module is used to predict the result of the token vector through a prediction function. The present invention can obtain the driver's driving status according to the driver's EEG and facial expression and give feedback in real time, which effectively improves the safety of driving.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to deep learning and intelligent biomedical signal processing, and more specifically, to a driver fatigue detection system based on a multimodal Transformer network. Background Art

[0002] Fatigue driving poses a huge traffic hazard, and the traffic accidents caused by it are too numerous to mention. According to road traffic industry survey data, among serious and major traffic accidents, accidents caused by fatigue driving account for more than 40%, which is one of the three major causes of serious and major traffic accidents, and accounts for as much as 21% of traffic accident deaths. If the driver's fatigue driving state can be detected in time, and then the automatic driving method is used to intervene, the occurrence of accidents can be reduced, thus saving many lives.

[0003] Existing driver fatigue detection models are mainly unimodal, either based on EEG or facial expression recognition.

[0004] Among them, EEG is realized based on brain-computer interface. Brain-computer interface refers to a direct link created between the human or animal brain and external devices to realize information exchange between the brain and the device. This technology can use the human brain to produce different reactions to different things or cognitive activities, thereby obtaining different types of EEG signals, and realize the conversion of control instructions by amplifying, filtering, collecting, feature extracting and classifying EEG signals.

[0005] Facial expressions play an important role in interpersonal communication. Compared with text and speech, facial expressions are more intuitive and accurate in expressing human emotions. Facial expression recognition methods generally include the following three aspects: face image preprocessing, facial expression feature learning and facial expression classification, and finally obtaining a person's physiological and psychological state based on facial expression classification.

[0006] Convolutional neural networks and recurrent neural networks are mainly used to realize the above-mentioned information processing of EEG or facial expressions. However, both convolutional neural networks and recurrent neural networks still have room for improvement in modeling the long-distance dependency of input features and parallel performance. However, for real-time detection systems, the inference time of the long-distance dependency model of input features will become a bottleneck restricting the performance of the model. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a driver fatigue detection system based on a multimodal Transformer network in view of the deficiencies in the prior art, thereby solving the problem that the inference time of the existing information processing network model restricts the model performance.

[0008] The driver fatigue detection system based on the multimodal Transformer network of the present invention comprises:

[0009] A data acquisition module, used for acquiring the driver's EEG signal and facial expression image, and dividing the facial expression image into facial expression image blocks;

[0010] An embedding processing module, used for embedding the EEG signal and the facial expression image block to obtain an EEG token and an image block token;

[0011] A self-attention analysis module, used for performing multi-head self-attention analysis on the EEG tokens and image block tokens through a Transformer encoder to obtain an EEG signal token vector and an image block token vector;

[0012] The result prediction module is used to predict the result of the EEG signal token vector and the image block token vector by using a prediction function that can perform dimensionality reduction processing.

[0013] The prediction function is:

[0014] prediction=tanh(W(Pooling([EEG class token; Image class token])))+b);

[0015] Where prediction is the predicted result value; EEG class token is the EEG signal token vector; Image class token is the image block token vector; Pooling is the pooling function; b is the bias vector; W is the weight vector.

[0016] A loss function is constructed according to the classification task of Transformer, and the predicted result value is analyzed by the loss function to optimize the predicted result value.

[0017] The loss function is,

[0018]

[0019] In the formula, yi indicates whether the predicted result value sample belongs to the i-th classification task; p i is the probability that the model believes that the sample belongs to the i-th classification task; task is the parameter factor; class is the number of categories.

[0020] The EEG signal token vector and the image block token vector are obtained by the following formula:

[0021] x'=MSA(LN(x))+x;

[0022] output=tanh(MLP(LN(x'))+x');

[0023] Where MSA is the multi-head self-attention mechanism function; LN is the layer normalization function; x is the input value of the Transformer encoder; x' is the output value of x after the multi-head self-attention mechanism function is operated; MLP is the multi-layer perceptron function; tanh is the classifier function; output is the output value of the Transformer encoder.

[0024] The EEG signal and facial expression image blocks are embedded using the following formula:

[0025] EEG=[EEG class ;EEG1;…;EEG t ]+pos EEG ;

[0026] Image=[Image class ;ImagePatch1;…;ImagePatch n ]+pos Image ;

[0027] Where EEG is the EEG token, and EEG∈R t×c ; R is the real number domain; t is the number of sampling points in the time dimension; C is the number of channels; EEG class is the EEG signal category token; pos EEG is the position embedding matrix of the EEG signal, and pos EEG ∈R (t+1)×c ; Image is the image block token; Image class is the facial expression image block category token; pos Image is the position embedding matrix of the facial expression image block, and pos Image ∈R (n+1)×c .

[0028] The position vectors are added to the EEG tokens and image patch tokens by the following formula,

[0029]

[0030]

[0031] Where, PE (pos,2i) is a row vector of elements; PE (pos,2i+1) is the column vector of elements; pos is the position of the EEG signal or facial expression image block in the time series, that is, the row vector in the position embedding matrix; i is the dimension of the matrix row vector.

[0032] The number of facial expression image blocks into which the facial expression image is segmented is,

[0033] n=H×L / P 2 ;

[0034] Where P is the image size; L is the image length; H is the image width; and C is the number of channels.

[0035] Beneficial Effects

[0036] The advantages of the present invention are: by fusing EEG signals and facial expression signals through an embedding processing module, a self-attention analysis module, and a result prediction module based on a Transformer module, the driving state of the driver can be obtained according to the driver's EEG and facial expressions and feedback can be given in real time, thereby solving the problem of the inference time restricting the model performance of the existing information processing network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a simplified flow chart of the detection system of the present invention;

[0038] Figure 2 It is a specific flow chart of the data acquisition module, the embedding processing module and the self-attention analysis module of the present invention;

[0039] Figure 3 Schematic diagram of the specific flow of the self-attention analysis module of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the embodiments, but it does not constitute any limitation to the present invention. Any limited number of modifications made by anyone within the scope of the claims of the present invention are still within the scope of the claims of the present invention.

[0041] See also Figure 1-Figure 3 The driver fatigue detection system based on the multimodal Transformer network of the present invention includes a data acquisition module, an embedding processing module, a self-attention analysis module, and a result prediction module. Among them, the embedding processing module, the self-attention analysis module, and the result prediction module are all implemented based on the Transformer module.

[0042] In this embodiment, the data acquisition module is used to collect the driver's EEG signals and facial expression image blocks.

[0043] Regarding the collection of EEG signals, this embodiment uses an EEG collection device to collect the driver's EEG signals to obtain a series of multi-channel EEG signals. Since the collection of EEG signals takes a certain time, the EEG signals can be used as a time series [EEG1, EEG2, ..., EEG t ]express.

[0044] A camera is placed behind the front windshield of the car to obtain the video stream of the driver. Since the collected EEG signal is a time series, it will have a certain duration. Therefore, the video frames of the video stream collected within the duration are averaged to obtain the average image of the driver's facial expression. The average image of the driver's facial expression will be cropped into n = H × L / P 2 facial expression image blocks, each of which has a size of C×P×P.

[0045] Among them, P is the image size; L is the image length; H is the image width; C is the number of channels.

[0046] The n facial expression image patches can be represented by a temporal sequence [ImagePatch1; ...; ImagePatchn].

[0047] The embedding processing module is used to embed the EEG signal and the facial expression image block to obtain EEG tokens and image block tokens. The EEG tokens and image block tokens are used as inputs of the multimodal Transformer model.

[0048] Regarding embedding processing, specifically, the formula for embedding the input using Transformer is as follows:

[0049] EEG=[EEG class ;EEG1;…;EEG t ]+pos EEG ;

[0050] Image=[Image class ;ImagePatch1;…;ImagePatch n ]+pos Image .

[0051] Where EEG is the EEG token, and EEG∈R t×c ; R is the real number domain; t is the number of sampling points in the time dimension; C is the number of channels; EEG class is the EEG signal category token, which is encoded by the Transformer encoder and then output to the multi-layer perceptron to classify the EEG signal; pos EEG is the position embedding matrix of the EEG signal, and pos EEG ∈R (t+1)×c ; Image is the image block token; Image class is the facial expression image block category token; pos Image is the position embedding matrix of the facial expression image block, and pos Image ∈R (n+1)×c .

[0052] Since the embedding of the input using Transformer has no order information, it is necessary to add a vector with order characteristics to each element in the token matrix. The position vector of the element in the token matrix can be determined by the following formula.

[0053]

[0054]

[0055] Where, PE (pos,2i) is a row vector of elements; PE (pos,2i+1) is the column vector of elements; pos is the position of the EEG signal or facial expression image block in the time series, that is, the row vector in the position embedding matrix; i is the dimension of the matrix row vector.

[0056] After embedding, the EEG signals and facial expression image blocks can make the position encoding of each signal or image block unique, and the distance between any two positions of time series of different lengths can be kept consistent. In addition, since trigonometric functions are used to encode the position of the signal or image block, the position encoding will not tend to infinity due to the infinite growth of the sequence.

[0057] The self-attention analysis module is used to perform self-attention analysis on the EEG tokens and image block tokens respectively through the Transformer encoder to obtain EEG signal token vectors and image block token vectors. The EEG signal token vector is used to represent the internal association of the EEG signal; the image block token vector is used to represent the association between facial expression image blocks.

[0058] Both the EEG signal token vector and the image block token vector can be obtained by the following formula:

[0059] X' = ​​MSA (LN (x)) + x;

[0060] output=tanh(MLP(LN(x'))+x').

[0061] In the formula, MSA is the multi-head self-attention mechanism function; LN is the layer normalization function; x is the input value of the encoder, which is EEG in the EEG signal classification task and Image in the image classification task; x' is the output value of x after the multi-head self-attention mechanism function is operated; MLP is the multi-layer perceptron function; tanh is the classifier function; output is the output value of the encoder, which is the EEG signal token vector or the image block token vector. After a series of encoder encoding analysis, the encoder input value allows the token vector to learn the internal association of the EEG signal and the association between image blocks.

[0062] The multi-head self-attention mechanism is described in detail below. The multi-head self-attention mechanism function can be specifically expressed by the following formula.

[0063]

[0064]

[0065] In the above formula, Q, K, and V are the three parameter factors of Query, Key, and Value in the multi-head self-attention mechanism.

[0066] The above three parameter factors can be obtained by multiplying the same input value by three different weight matrices:

[0067] Q=W q ×Input;

[0068] K=W k ×Input;

[0069] V=W v ×Input.

[0070] Among them, W q , W k , W v Both are weight matrices; Input is the input value, equal to x.

[0071] By calculating the correlation between Q and K, a set of attention weights can be obtained. The attention weights are mapped to the interval [0, 1] through the softmax function. However, when the dimension of the vector is very high, QK T It will reach a very large magnitude, which makes the gradient of the softmax function very small, so it is necessary to T Based on , we divide it by a parameter dk and normalize it in dimension to eliminate the influence of dimension.

[0072] In this embodiment, in order to improve the parallel performance of the model, the input EEG signal or average image can be divided into multiple attention heads through a multi-head self-attention mechanism. That is, the input EEG signal or image block matrix is ​​split into multiple parts. The results of each part after the self-attention mechanism operation are spliced ​​to improve the parallel performance of the model.

[0073] The result prediction module is used to predict the results of the EEG signal token vector and the image block token vector through a prediction function that can perform dimensionality reduction processing after obtaining the token vector.

[0074] The specific prediction method can be achieved through the following functional formula:

[0075] prediction=tanh(W(Pooling([EEG class token; Image class token]))+b).

[0076] In the formula, prediction is the predicted result value; EEG class token is the EEG signal token vector; Imageclasstoken is the image block token vector; Pooling is the pooling function used to reduce the dimension of the token vector; b is the bias vector; W is the weight vector. Among them, the two parameters W and b together constitute the fully connected layer, which performs a linear transformation on the vector passing through the fully connected layer.

[0077] The Transformer network originates from the field of natural language processing, and its core is to learn the relationship between elements at different positions of a sequence through a module called the self-attention mechanism. Compared with convolutional neural networks, the Transformer network has a more flexible receptive field and can better capture long-distance dependencies in feature maps; compared with recurrent neural networks, the Transformer network does not need to wait for the output of the previous state as the input of the model, and has more superior parallel performance. Therefore, this embodiment can fuse the information of the two modalities of EEG and facial expression through the Transformer network technology, and can obtain the driver's driving status based on the driver's EEG and facial expressions and give feedback in real time, thereby solving the problem that the inference time of the existing information processing network model restricts the performance of the model. In addition, the Transformer model without convolution operation will perform better than the convolutional neural network in terms of parameter quantity and inference time.

[0078] Preferably, this embodiment also designs a loss function so that the Transformer module can directly use the encoder to process visual features and can interact with multimodal features without adding an additional visual encoder.

[0079] Since the Transformer model is composed of three classification tasks, the loss function formula of this embodiment is designed as follows:

[0080]

[0081] Among them, y i Indicates whether the predicted result value sample belongs to the i-th classification task; p i is the probability that the model believes that the sample belongs to the i-th classification task; task is the parameter factor; class is the number of categories.

[0082] By analyzing the predicted result value through the loss function, the predicted result value can be optimized and the Transformer model can more accurately feedback the driver's status.

[0083] When i =1, p i The closer it is to 1, the smaller the penalty, that is, the smaller the gradient transmitted when the sample is back propagated. i The closer it is to 0, the greater the penalty, that is, the greater the gradient passed when the sample is backpropagated.

[0084] The above is only a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the structure of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. Driver fatigue detection system based on multimodal Transformer network, characterized by: include, A data acquisition module, used for acquiring the driver's EEG signal and facial expression image, and dividing the facial expression image into facial expression image blocks; An embedding processing module, used for embedding the EEG signal and the facial expression image block to obtain an EEG token and an image block token; A self-attention analysis module, used for performing multi-head self-attention analysis on the EEG tokens and image block tokens through a Transformer encoder to obtain an EEG signal token vector and an image block token vector; A result prediction module, used for predicting the result of the EEG signal token vector and the image block token vector by using a prediction function capable of performing dimensionality reduction processing; The EEG signal and facial expression image blocks are embedded using the following formula: EEG=[EEG class ;EEG1;…;EEG t ]+pos EEG ; Image=[Image class ;ImagePatch1;…;ImagePatch n ]+pos Image ; Where EEG is the EEG token, and EEG∈R t×c ; R is the real number domain; t is the number of sampling points in the time dimension; C is the number of channels; EEG class is the EEG signal category token; pos EEG is the position embedding matrix of the EEG signal, and pos EEG ∈R (t +1)×c ; Image is the image block token; Image vlass is the facial expression image block category token; pos Image is the position embedding matrix of the facial expression image block, and pos Image ∈R (n+1)×c ; The position vectors are added to the EEG tokens and image patch tokens by the following formula, Where, PE (pos,2i) is a row vector of elements; PE (pos,2i+1) is the column vector of elements; pos is the position of the EEG signal or facial expression image block in the time series, that is, the row vector in the position embedding matrix; i is the dimension of the matrix row vector; The number of facial expression image blocks into which the facial expression image is segmented is, n=H×L / P 2 ; Where P is the image size; L is the image length; H is the image width; and C is the number of channels.

2. The driver fatigue detection system based on multimodal Transformer network according to claim 1 is characterized in that: The prediction function is: prediction=tanh(W(Pooling([EEGclasstoken;Imageclasstoken]))+b); Where prediction is the predicted result value; EEG class token is the EEG signal token vector; Image class token is the image block token vector; Pooling is the pooling function; b is the bias vector; W is the weight vector.

3. The driver fatigue detection system based on multimodal Transformer network according to claim 2 is characterized in that: A loss function is constructed according to the classification task of Transformer, and the predicted result value is analyzed by the loss function to optimize the predicted result value.

4. The driver fatigue detection system based on multimodal Transformer network according to claim 3 is characterized in that: The loss function is, In the formula, y i Indicates whether the predicted result value sample belongs to the i-th classification task; p i is the probability that the model believes that the sample belongs to the i-th classification task; task is the parameter factor; class is the number of categories.

5. The driver fatigue detection system based on multimodal Transformer network according to claim 1 is characterized in that: The EEG signal token vector and the image block token vector are obtained by the following formula: x'=MSA(LN(x))+x; output=tanh(MLP(LN(x'))+x'); Where MSA is the multi-head self-attention mechanism function; LN is the layer normalization function; x is the input value of the Transformer encoder; x' is the output value of x after the multi-head self-attention mechanism function is operated; MLP is the multi-layer perceptron function; tanh is the classifier function; output is the output value of the Transformer encoder.

Citation Information

Cited By

  • Rapid fatigue detection method and device based on electroencephalogram characteristics

    CN120078415A