Fatigue driving detection method and system based on multi-modal feature fusion

Abstract

The invention discloses a fatigue driving detection method and system based on multi-modal feature fusion, and the method is explained by taking a DROZY data set as an embodiment. Through preprocessing a data set, image data and PSG time sequence data which are in one-to-one correspondence in time are extracted, and the image data and the time sequence data are input and used by a subsequent system. Feature information of shallow, medium and deep networks is extracted from the two types of modal data; and after the dimensions of the three feature blocks of the two modes are adjusted to be consistent, feature information is optimized through a residual convolution attention module. And obtaining weight information of different layers of the single-mode data through a weight calculation method, and carrying out multi-scale weight fusion on the weight information and an original feature block to obtain a feature vector extraction result of the image data and the time series data. And fusing the feature vector data of the two modes, and outputting a classification result through a classifier according to the fused features to realize fatigue driving detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and relates to a fatigue driving detection method and system based on multimodal feature fusion. Background Art

[0002] Fatigue or sleepiness is a pervasive mental state that has a profound impact on daily life, especially during tasks that require sustained attention and alertness. When fatigue affects driving, it can have serious consequences. The condition causes reduced brain activity, slower reaction times, and impaired judgment. Driving requires sustained focus, and driving while fatigued can lead to decreased reaction times, misjudgments, and even catastrophic accidents.

[0003] Although there are some fatigue detection indicators, such as the percentage of time eyes closed (PERCLOS) and heart rate variability, it is still difficult to accurately quantify the degree of fatigue. The PERCLOS value is affected by factors such as lighting and facial angle, and there will be errors. In addition, different individuals have different fatigue thresholds, making it difficult to determine a unified standard to accurately measure whether fatigue is mild, moderate or severe. For example, the same PERCLOS value may only be a normal blink for a young driver who gets enough sleep, but it may mean fatigue for an elderly driver who does not get enough sleep. Due to individual differences, sensor errors, and environmental factors, fatigue driving detection systems are prone to misjudgment. For example, when a driver normally checks the rearview mirror or thinks about a problem briefly, his eyes may be closed for a longer time or his head posture may change, which may be misjudged as fatigue driving. When a vehicle is driving on a road with many curves, speed changes and lane deviations are normal driving operations, but they may be mistakenly judged as fatigue driving.

[0004] Traditional detection methods have the disadvantages of relying on manual experience and time-consuming target feature selection and debugging, which leads to a decrease in judgment accuracy when the data sample size is small. Therefore, how to optimize the traditional feature extraction method, use deep learning technology to extract data features, and use this multimodal data for data fusion to improve the accuracy and efficiency of judging driving fatigue is a practical challenge. Summary of the invention

[0005] The purpose of the present invention is to solve the problem of low accuracy of fatigue driving detection in the prior art and to provide a fatigue driving detection method and system based on multimodal feature fusion.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention proposes a fatigue driving detection method based on multimodal feature fusion, which comprises the following steps:

[0008] Obtain shallow, mid-level, and deep features of multimodal data;

[0009] Process the shallow, middle and deep features of multimodal data to obtain the weights of feature blocks at each layer;

[0010] The weights of the feature blocks of each layer are fused to obtain the final multimodal feature vector;

[0011] The final multimodal feature vector is processed to obtain the fatigue driving classification result and realize fatigue driving detection.

[0012] Preferably, the shallow features, middle features and deep features of the multimodal data are obtained as follows:

[0013] Acquire PSG time series signal data and pre-processed image data; wherein the multimodal data includes image data and PSG time series signal data;

[0014] The ResNet18 neural network model is used to extract the feature information of the initial shallow, medium and deep networks of the preprocessed image data;

[0015] The LSTM neural network is used to extract the feature information of the initial shallow, medium and deep networks of the PSG time series signal data;

[0016] The initial shallow features of the PSG time series signal data and the preprocessed image data are subjected to dimensionality reduction processing, and the initial deep features of the PSG time series signal data and the preprocessed image data are subjected to dimensionality increase processing. The dimensions of the initial shallow features and the initial deep features are consistent with the dimensions of the initial middle features, so as to obtain the shallow features, middle features and deep features of the multimodal data.

[0017] Preferably, the PSG timing signal data obtained includes EEG data, EMG data, EOG data, and ECG data;

[0018] Face recognition is used to extract facial images frame by frame to obtain facial image data, which is the preprocessed image data.

[0019] Preferably, the shallow features, middle features and deep features of the multimodal data are processed to obtain the weights of the feature blocks of each layer, specifically:

[0020] The residual convolutional attention module is used to optimize the feature information of the shallow features, middle features and deep features of the single modality respectively; after the feature optimization is completed through the residual convolutional attention module, the weight calculation method is used to determine the weight of the feature blocks of each layer; among them, the residual convolutional attention module connects the channel attention module and the spatial attention module, and adds residuals in the attention module to prevent the gradient from disappearing.

[0021] Preferably, the optimized feature information is globally pooled and then convolved with a 1*1 convolution kernel. The result is divided by the output of the spatial attention module to obtain three redistributed weight values, which are normalized to obtain the weights of the feature blocks of each layer.

[0022] Preferably, the weights of the feature blocks of each layer are fused to obtain the final multimodal feature vector, specifically:

[0023] The weights of the feature blocks of each layer of each modality are fused, and then the fused feature vectors of each modality are further fused through the Hadamard product to obtain the final multimodal feature vector.

[0024] The present invention proposes a fatigue driving detection system based on multimodal feature fusion, comprising:

[0025] A feature parameter acquisition module, which is used to acquire shallow features, middle features and deep features of multimodal data;

[0026] A weight parameter acquisition module, which is used to process shallow features, middle features and deep features of multimodal data to obtain weights of feature blocks at each layer;

[0027] A data fusion module, which is used to fuse the weights of feature blocks at each layer to obtain a final multimodal feature vector;

[0028] The classification result acquisition module is used to process the final multimodal feature vector, obtain the fatigue driving classification result, and realize fatigue driving detection.

[0029] Preferably, the shallow features, middle features and deep features of the multimodal data are obtained as follows:

[0030] Acquire PSG time series signal data and pre-processed image data; wherein the multimodal data includes image data and PSG time series signal data;

[0031] The ResNet18 neural network model is used to extract the feature information of the initial shallow, medium and deep networks of the preprocessed image data;

[0032] The LSTM neural network is used to extract the feature information of the initial shallow, medium and deep networks of the PSG time series signal data;

[0033] The initial shallow features of the PSG time series signal data and the preprocessed image data are subjected to dimensionality reduction processing, and the initial deep features of the PSG time series signal data and the preprocessed image data are subjected to dimensionality increase processing. The dimensions of the initial shallow features and the initial deep features are consistent with the dimensions of the initial middle features, so as to obtain the shallow features, middle features and deep features of the multimodal data.

[0034] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a fatigue driving detection method based on multimodal feature fusion when executing the computer program.

[0035] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a fatigue driving detection method based on multimodal feature fusion.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention proposes a fatigue driving detection method based on multimodal feature fusion. By preprocessing the data set, the facial images of the experimental object in the fatigue and non-fatigue states are extracted frame by frame, and the PSG time series signal data of the corresponding time are sorted out. The image and time series data are used for subsequent system input. The feature information of the shallow, medium and deep networks are extracted for the two modal data respectively, and the feature blocks of the 3rd, 9th and 18th layers of the network are extracted. After the dimensions of the three feature blocks of the two modalities are adjusted to be consistent, the feature information is optimized. Then, the weight information of different layers of the single modal data is obtained by the weight calculation method, and then the weight information is multi-scale weighted fused with the original feature block to obtain the feature vector extraction results of the image data and the time series data. The feature vector data of the two modalities are Hadamard product for feature fusion, and then the classification results are output by the fused multimodal features through a fully connected neural network classifier to realize fatigue driving detection. The method can process multimodal data of images and time series, extract feature information of different network depths, and perform multimodal data feature fusion to provide driving fatigue detection results with higher accuracy.

[0038] The present invention proposes a fatigue driving detection system based on multimodal feature fusion, which divides the system into a feature parameter acquisition module, a weight parameter acquisition module, a data fusion module and a classification result acquisition module, obtains fatigue driving classification results, and realizes fatigue driving detection. The modularization concept is adopted to make each module independent of each other, which is convenient for unified management of each module. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 This is a flow chart of the fatigue driving detection method based on multimodal feature fusion of the present invention.

[0041] Figure 2 This is a diagram of the DROZY data set preprocessing structure of the present invention.

[0042] Figure 3 This is a diagram of the facial image data preprocessing idea of ​​the present invention.

[0043] Figure 4 This is the ResNet18 structure diagram of the present invention.

[0044] Figure 5 This is the LSTM structure diagram of the present invention.

[0045] Figure 6 This is the structural diagram of the residual convolution attention module of the present invention.

[0046] Figure 7 It is a structural diagram of the adaptive weight calculation method of the present invention.

[0047] Figure 8 It is a structural diagram of the Hadamard product feature fusion method of the present invention.

[0048] Fig. 9 This is a structural diagram of fatigue driving detection based on a multimodal feature fusion model of the present invention.

[0049] Fig.10 This is a diagram of the fatigue driving detection system based on multimodal feature fusion of the present invention.

[0050] Fig.11 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0054] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0055] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0056] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0057] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0058] Example 1

[0059] The present invention proposes a fatigue driving detection method based on multimodal feature fusion, such as Figure 1 As shown, including:

[0060] S1, obtain shallow features, middle features and deep features of multimodal data;

[0061] The shallow features, middle features and deep features of the multimodal data are obtained as follows:

[0062] Acquire PSG time series signal data and pre-processed image data; wherein the multimodal data includes image data and PSG time series signal data;

[0063] The ResNet18 neural network model is used to extract the feature information of the initial shallow, medium and deep networks of the preprocessed image data;

[0064] The LSTM neural network is used to extract the feature information of the initial shallow, medium and deep networks of the PSG time series signal data;

[0065] The initial shallow features of the PSG time series signal data and the preprocessed image data are subjected to dimensionality reduction processing, and the initial deep features of the PSG time series signal data and the preprocessed image data are subjected to dimensionality increase processing. The dimensions of the initial shallow features and the initial deep features are consistent with the dimensions of the initial middle features, so as to obtain the shallow features, middle features and deep features of the multimodal data.

[0066] Acquire PSG time series signal data including EEG data, EMG data, EOG data, and ECG data;

[0067] Face recognition is used to extract facial images frame by frame to obtain facial image data, which is the preprocessed image data.

[0068] S2, processing the shallow features, middle features and deep features of the multimodal data to obtain the weights of the feature blocks of each layer;

[0069] The shallow features, middle features and deep features of the multimodal data are processed to obtain the weights of the feature blocks of each layer, specifically:

[0070] The residual convolutional attention module is used to optimize the feature information of the shallow features, middle features and deep features of the single modality respectively; after the feature optimization is completed through the residual convolutional attention module, the weight calculation method is used to determine the weight of the feature blocks of each layer; among them, the residual convolutional attention module connects the channel attention module and the spatial attention module, and adds residuals in the attention module to prevent the gradient from disappearing.

[0071] The optimized feature information is globally pooled and then convolved with a 1*1 convolution kernel. The result is divided by the output of the spatial attention module to obtain three redistributed weight values, which are normalized to obtain the weights of the feature blocks of each layer.

[0072] S3, fusing the weights of feature blocks at each layer to obtain the final multimodal feature vector;

[0073] The weights of the feature blocks of each layer are fused to obtain the final multimodal feature vector, which is specifically:

[0074] The weights of the feature blocks of each layer of each modality are fused, and then the fused feature vectors of each modality are further fused through the Hadamard product to obtain the final multimodal feature vector.

[0075] S4. Process the final multimodal feature vector to obtain fatigue driving classification results and realize fatigue driving detection.

[0076] The following is a detailed description of this method:

[0077] This paper proposes a fatigue driving detection method based on multimodal feature fusion. Figure 2 As shown, the video data and PSG time series signal data are selected. For the video data, a face recognition frame-by-frame facial image processing method is adopted to obtain face image data. For the PSG time series data, four modal data are selected: EEG data, EMG data, EOG data, and ECG data. The data part corresponding to the face image data at the moment is extracted, and the irrelevant data part is eliminated.

[0078] For the preprocessing details of facial image data, Figure 3 As shown in the figure, the original data provided by the DROZY dataset is continuous image data. The image data is extracted frame by frame and converted into picture data. Then, the face information in each picture data is intercepted using the face recognition algorithm of the OpenCV Dlib library. The Dlib library uses deep learning technology to achieve face recognition tasks through CNN face detection, face key point detection, DCNN face feature extraction, feature comparison and recognition steps. By mapping the face image to a high-dimensional feature vector space and comparing the distance or similarity between different feature vectors, accurate face recognition can be achieved. Then, the face image pixel size is reset to 75*75, and the preprocessing operation of the face image data is completed.

[0079] For the processed facial image, select a suitable deep learning network to extract features and encode it with reference to the ResNet18 neural network model. Figure 4As shown, ResNet18 is a variant of the deep residual network, which is a deep learning model architecture proposed by Microsoft Research. The present invention applies the ResNet18 model, and controls the input layer dimension to be (n, 1, 75, 75), where n represents the size of the processing batch, 1 represents that the input data is a grayscale image, and 75 and 75 represent the height and width of the input image. The output layer dimension is controlled to be a two-dimensional tensor of (n, 256), where n represents the batch size, and 256 is the feature dimension of the output feature, which is used for subsequent classification tasks. The model includes 18 convolutional layers, which are used to extract feature information of the input image, and each convolutional layer is followed by batch normalization (BN) and activation function (ReLU). A maximum pooling layer is added after the first convolutional layer of the model to reduce the spatial dimension of the feature map by taking the maximum value of the local area, and retain the most significant feature information of the input image, which has an important impact on the feature extraction of subsequent layers. An average pooling layer is added after the last convolutional layer of the model and before the fully connected layer. The feature information is aggregated by taking the average value of the local area, reducing the complexity of the model and reducing the risk of overfitting. The key point of the model is the introduction of residual connections, which transmit information through direct connections across layers. Each residual module includes two convolutional layers. The input of the module is processed through these layers and then added to the original input to form the output. The model contains 8 residual modules. In the residual module, the input of a layer in the network can be directly passed to the subsequent layer by skipping some layers, so that the network can learn the residual function instead of learning the original mapping function. This design can effectively alleviate the gradient vanishing problem and enable the network to be effectively trained at deeper layers.

[0080] The preprocessing of PSG time series information data is to remove the parts that do not correspond to the facial image data selected for use, and integrate the originally independent EEG, EMG, EOG, and ECG data into multi-dimensional time series data. Select a deep learning network suitable for processing time series signals to extract feature information, and use LSTM to encode time series data, such as Figure 5As shown. LSTM is a special recurrent neural network, which consists of a cell state and three gated units: forget gate, input gate and output gate. The forget gate determines the information forgotten in the cell state, and generates a value of 0 to 1 by operating the input and the hidden state of the previous moment, which is used to control the degree of information retention in the cell state. The input gate determines the input information added to the cell state, and generates a value of 0 to 1 by operating the input and the hidden state of the previous moment, which is used to control the addition of new information. The output gate determines the output of the hidden state at the current moment, and generates a value of 0 to 1 by operating the input and the hidden state of the previous moment, which is used to control the range of the output. In order to obtain the time series feature data corresponding to the image feature extraction model ResNet18, the present invention designs a deep network structure containing 18 LSTM units. These LSTM units are connected by splicing to effectively capture the complex patterns and dependencies in the input time series data, and form a corresponding relationship between the feature information of the time series data and the feature information of the image data. The input layer dimension of the LSTM model is controlled to be (n, 1280, 10), where n represents the batch size, 1280 is the extracted time step, and 10 is the feature dimension extracted for each time step. The output layer dimension of the control model is a two-dimensional tensor of (n, 256), where n represents the batch size and 256 is the feature dimension of the output feature, which is used for subsequent classification tasks.

[0081] Through the ResNet18 network and LSTM network, the extracted 3-layer, 9-layer, and 18-layer image feature blocks and time series feature blocks are dimensionally adjusted, and the 3-layer feature block is reduced in dimension and the 18-layer feature block is increased in dimension, so that the feature block dimension is consistent with the 9-layer feature block. Figure 6 As shown in the figure, the residual convolution attention module (RCAM) is used to optimize the feature information of the three feature blocks of the single modality. The residual convolution attention module connects the channel attention module (CAM) and the spatial attention module (SAM), and adds residuals to the attention module to prevent the gradient from disappearing.

[0082]

[0083] O a =SAM(I+O c )+I

[0084] For the Oc equation, Oc is the output of the channel attention module. CAM() is used to generate a category activation map to help understand the model decision area. ReLU() is used as the activation function, C is the number of channels, and N is the spatial dimension of the feature block. X is the input item of the RCAM module, W is the initial weight, and b is the bias.

[0085]

[0086] In CAM(), ω i represents the weight of channel i, A i represents the activation value of feature map A on channel i, and C represents the number of channels of feature map A.

[0087] For the Oa equation, Oa is the output of the spatial attention module. SAM() enhances the important areas in the feature map by calculating spatial attention. I is the input feature block of the RCAM module, and Oc is the output of the channel attention module.

[0088] SAM=S☉F

[0089] In SAM(), F represents the input feature block and S represents the spatial attention map obtained by generating the attention map.

[0090] S=o(W s ·F concat )

[0091] σ is the activation function, usually sigmoid(). s is the weight of the convolution kernel, F concat It is the concatenation result of average pooling and maximum pooling in the channel dimension.

[0092] F concat =[F avg , F max ]

[0093] F avg =AgPool(F)

[0094] F max =MaxPool(F)

[0095] As shown above, F avg is the average pooling, F max is the maximum pooling.

[0096] By using the residual convolution attention module, the present invention can significantly enhance the ability to extract and express feature information. After completing feature optimization through the residual convolution attention module, the weight calculation method is used to determine the weight of each layer of feature blocks. Figure 7 As shown in the figure, the feature block after RCAM optimization is subjected to global pooling (GAP) processing to obtain a one-dimensional vector, which is then convolved with a 1*1 convolution kernel to avoid overfitting of the model. The result is divided by Oa itself to obtain the weight of the unimodal feature block Ow.

[0097]

[0098] In the equation, Ow is the weight value of the unimodal feature block, Oa is the feature block output by the RCAM module, Conv() is a 1*1 convolution kernel operation, H is the number of rows of the Oa feature map, and W is the number of columns of the Oa feature map.

[0099] The same operation is performed on the other two feature blocks to finally obtain three redistributed weight values ​​Ow, and classification calculation is performed to obtain the weights W of the feature blocks of the 3rd, 9th, and 18th layers.

[0100]

[0101] Where W i is the weight value of the feature block in the i-th layer. is the weight value of the unimodal feature block of the i-th layer.

[0102] The weight values ​​of the three feature blocks corresponding to the image mode and the time series mode are obtained for weight fusion. Then, the feature blocks of the image data and the time series data after weight fusion are adjusted using the classifier to obtain an n-dimensional feature vector. Figure 8 As shown, the image modality feature vector W containing weight information I *I and time series modal eigenvector W T *T performs further feature fusion operation through Hadamard product, i.e. one-to-one multiplication, to obtain the final multimodal feature vector W*I*T of fused image and time series feature information. The multimodal feature vector is input into the fully connected neural network classifier to process the fusion features, and the final classification result is obtained through iterative training.

[0103] The present invention adopts a fusion method in the form of Hadamard product when performing feature fusion. Different from several common fusion methods, this is an element-level fusion method. Element-level fusion retains feature information without introducing changes in feature dimensions, and through the interaction of element levels, it can capture the correlation between different modal features, which helps to improve the expressive power of the fused features, and thus improve the accuracy of model prediction. Element-level fusion in the form of Hadamard product is a more appropriate feature fusion method selected by the present invention in practical applications after comprehensively considering task requirements, data characteristics and fusion effects.

[0104] For the coupled feature data, a fully connected classifier is used to output the model classification judgment results. Through the iterative training of the model using deep learning technology, the model finally outputs the detection results of fatigue driving.

[0105] The above is the macro level, the construction idea of ​​the fatigue driving detection method based on the deep learning multimodal feature fusion model proposed by the present invention. At the same time, there are many details that need to be paid attention to when constructing this fatigue driving detection model.

[0106] The present invention proposes a fatigue driving detection method based on a multimodal feature fusion model, the process of which is as follows: Fig. 9 As shown, the following steps are included:

[0107] Preprocess the data set to sort out the image data information and time series data information corresponding to the time;

[0108] According to the category of multimodal data, use appropriate deep learning networks, i.e. ResNet18 for image data and LSTM model for time series data to extract shallow, middle and deep feature information. Reduce the dimension of shallow features and increase the dimension of deep features to keep their dimensions consistent with those of middle features;

[0109] For the three feature blocks of the image modality and the three feature blocks of the time series modality, the residual convolution attention module is used to optimize the feature information, and then the weights of the feature blocks of each layer are determined by the weight calculation method;

[0110] The three feature blocks of the image modality and the three feature blocks of the time series modality are weighted and fused respectively. Then, the feature blocks obtained after weight fusion of the two modalities are dimensionalized using a classifier to obtain an n-dimensional feature vector. Then, the image modality feature vector W containing the weight information is converted into I *I and time series modal eigenvector W T *T performs further feature fusion operation through Hadamard product, that is, one-to-one corresponding multiplication, to obtain the final multimodal feature vector W*I*T of fused image and time series feature information;

[0111] The fused features are processed by a fully connected neural network classifier, and the prediction results of fatigue driving are obtained through iterative training.

[0112] In summary, the present invention provides a fatigue driving detection method based on a multimodal feature fusion model, which couples multimodal feature information to build a scientific and reasonable deep learning model. The present invention provides valuable ideas and feasible solutions for fatigue driving detection. By multi-scale fusion of single-modal data and performing Hadamard product on the element-level coupling of multimodal feature vectors, the present invention has greatly helped improve the accuracy of fatigue driving detection. The present invention has the following advantages:

[0113] 1) Multimodal fatigue data simulation: The fatigue data of the present invention selects facial image data, EEG data, EMG data, EOG data, and ECG data. The data covers a wide range and is closely related to brain fatigue driving at the physiological level, providing a detailed and accurate simulation of the fatigue driving state.

[0114] 2) Deep learning model with strong versatility: The present invention uses the DROZY fatigue database to train the deep learning fatigue driving detection model. The generated deep learning neural network model can obtain higher accuracy when processing the same type of fatigue data. At the same time, based on the original model, the transfer learning retraining model can obtain considerable detection accuracy, which has high versatility in fatigue driving detection scenarios.

[0115] 3) Quantitative judgment of fatigue status: Using the KSS scale as a reference indicator, the fatigue driving detection method based on the deep learning multimodal feature fusion model proposed in the present invention can not only qualitatively judge whether the driver is fatigued or not, but also evaluate the driver's current fatigue level.

[0116] Example 2

[0117] The present invention proposes a fatigue driving detection system based on multimodal feature fusion, such as Fig.10 As shown, including:

[0118] A feature parameter acquisition module, which is used to acquire shallow features, middle features and deep features of multimodal data;

[0119] The shallow features, middle features and deep features of the multimodal data are obtained as follows:

[0120] Acquire PSG time series signal data and pre-processed image data; wherein the multimodal data includes image data and PSG time series signal data;

[0121] The ResNet18 neural network model is used to extract the feature information of the initial shallow, medium and deep networks of the preprocessed image data;

[0122] The LSTM neural network is used to extract the feature information of the initial shallow, medium and deep networks of the PSG time series signal data;

[0123] The initial shallow features of the PSG time series signal data and the preprocessed image data are subjected to dimensionality reduction processing, and the initial deep features of the PSG time series signal data and the preprocessed image data are subjected to dimensionality increase processing. The dimensions of the initial shallow features and the initial deep features are consistent with the dimensions of the initial middle features, so as to obtain the shallow features, middle features and deep features of the multimodal data.

[0124] Acquire PSG time series signal data including EEG data, EMG data, EOG data, and ECG data;

[0125] Face recognition is used to extract facial images frame by frame to obtain facial image data, which is the preprocessed image data.

[0126] A weight parameter acquisition module, which is used to process shallow features, middle features and deep features of multimodal data to obtain weights of feature blocks at each layer;

[0127] The shallow features, middle features and deep features of the multimodal data are processed to obtain the weights of the feature blocks of each layer, specifically:

[0128] The residual convolutional attention module is used to optimize the feature information of the shallow features, middle features and deep features of the single modality respectively; after the feature optimization is completed through the residual convolutional attention module, the weight calculation method is used to determine the weight of the feature blocks of each layer; among them, the residual convolutional attention module connects the channel attention module and the spatial attention module, and adds residuals in the attention module to prevent the gradient from disappearing.

[0129] The optimized feature information is globally pooled and then convolved with a 1*1 convolution kernel. The result is divided by the output of the spatial attention module to obtain three redistributed weight values, which are normalized to obtain the weights of the feature blocks of each layer.

[0130] A data fusion module, which is used to fuse the weights of feature blocks at each layer to obtain a final multimodal feature vector;

[0131] The weights of the feature blocks of each layer are fused to obtain the final multimodal feature vector, which is specifically:

[0132] The weights of the feature blocks of each layer of each modality are fused, and then the fused feature vectors of each modality are further fused through the Hadamard product to obtain the final multimodal feature vector.

[0133] The classification result acquisition module is used to process the final multimodal feature vector, obtain the fatigue driving classification result, and realize fatigue driving detection.

[0134] Example 3

[0135] See also Fig.11 As shown, the present invention also provides an electronic device 100 for a fatigue driving detection method based on a multimodal feature fusion model; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0136] The memory 101 can be used to store the computer program 103, and the processor 102 implements the steps of the fatigue driving detection method based on the multimodal feature fusion model described in Example 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 can include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0137] The at least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.

[0138] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a fatigue driving detection method based on a multimodal feature fusion model, and the processor 102 can execute the plurality of instructions to implement:

[0139] Obtain shallow, mid-level, and deep features of multimodal data;

[0140] Process the shallow, middle and deep features of multimodal data to obtain the weights of feature blocks at each layer;

[0141] The weights of the feature blocks of each layer are fused to obtain the final multimodal feature vector;

[0142] The final multimodal feature vector is processed to obtain the fatigue driving classification result and realize fatigue driving detection.

[0143] Example 4

[0144] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0145] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A fatigue driving detection method based on multimodal feature fusion, characterized in that: The following steps are involved: Obtain shallow, mid-level, and deep features of multimodal data; Process the shallow, middle and deep features of multimodal data to obtain the weights of feature blocks at each layer; The weights of the feature blocks of each layer are fused to obtain the final multimodal feature vector; The final multimodal feature vector is processed to obtain the fatigue driving classification result and realize fatigue driving detection.

2. The fatigue driving detection method based on multimodal feature fusion according to claim 1 is characterized in that: The shallow features, middle features and deep features of the multimodal data are obtained as follows: Acquire PSG time series signal data and pre-processed image data; wherein the multimodal data includes image data and PSG time series signal data; The ResNet18 neural network model is used to extract the feature information of the initial shallow, medium and deep networks of the preprocessed image data; The LSTM neural network is used to extract the feature information of the initial shallow, medium and deep networks of the PSG time series signal data; The initial shallow features of the PSG time series signal data and the preprocessed image data are subjected to dimensionality reduction processing, and the initial deep features of the PSG time series signal data and the preprocessed image data are subjected to dimensionality increase processing. The dimensions of the initial shallow features and the initial deep features are consistent with the dimensions of the initial middle features, so as to obtain the shallow features, middle features and deep features of the multimodal data.

3. The fatigue driving detection method based on multimodal feature fusion according to claim 2 is characterized in that: Acquiring PSG time series signal data including EEG data, EMG data, EOG data, and ECG data; Face recognition is used to extract facial images frame by frame to obtain facial image data, which is the preprocessed image data.

4. The fatigue driving detection method based on multimodal feature fusion according to claim 1 is characterized in that: The shallow features, middle features and deep features of the multimodal data are processed to obtain the weights of the feature blocks of each layer, specifically: The residual convolutional attention module is used to optimize the feature information of the shallow features, middle features and deep features of the single modality respectively; after the feature optimization is completed through the residual convolutional attention module, the weight calculation method is used to determine the weight of the feature blocks of each layer; among them, the residual convolutional attention module connects the channel attention module and the spatial attention module, and adds residuals in the attention module to prevent the gradient from disappearing.

5. The fatigue driving detection method based on multimodal feature fusion according to claim 4 is characterized in that: The optimized feature information is globally pooled and then convolved with a 1*1 convolution kernel. The result is divided by the output of the spatial attention module to obtain three redistributed weight values, which are normalized to obtain the weights of the feature blocks of each layer.

6. The fatigue driving detection method based on multimodal feature fusion according to claim 1 is characterized in that: The weights of the feature blocks of each layer are fused to obtain the final multimodal feature vector, which is specifically: The weights of the feature blocks of each layer of each modality are fused, and then the fused feature vectors of each modality are further fused through the Hadamard product to obtain the final multimodal feature vector.

7. A fatigue driving detection system based on multimodal feature fusion, characterized in that: include: A feature parameter acquisition module, which is used to acquire shallow features, middle features and deep features of multimodal data; A weight parameter acquisition module, which is used to process shallow features, middle features and deep features of multimodal data to obtain weights of feature blocks at each layer; A data fusion module, which is used to fuse the weights of feature blocks at each layer to obtain a final multimodal feature vector; The classification result acquisition module is used to process the final multimodal feature vector, obtain the fatigue driving classification result, and realize fatigue driving detection.

8. The fatigue driving detection system based on multimodal feature fusion according to claim 7 is characterized in that: The shallow features, middle features and deep features of the multimodal data are obtained as follows: Acquire PSG time series signal data and pre-processed image data; wherein the multimodal data includes image data and PSG time series signal data; The ResNet18 neural network model is used to extract the feature information of the initial shallow, medium and deep networks of the preprocessed image data; The LSTM neural network is used to extract the feature information of the initial shallow, medium and deep networks of the PSG time series signal data; The initial shallow features of the PSG time series signal data and the preprocessed image data are subjected to dimensionality reduction processing, and the initial deep features of the PSG time series signal data and the preprocessed image data are subjected to dimensionality increase processing. The dimensions of the initial shallow features and the initial deep features are consistent with the dimensions of the initial middle features, so as to obtain the shallow features, middle features and deep features of the multimodal data.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the fatigue driving detection method based on multimodal feature fusion as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the fatigue driving detection method based on multimodal feature fusion as described in any one of claims 1 to 6 are implemented.

Images