A fatigue driving detection method based on dual-source multi-scale feature fusion network

Abstract

The present invention discloses a fatigue driving detection method based on a dual-source multi-scale feature fusion network. The method performs frequency division filtering on the collected EEG signals and extracts differential entropy as channel features, and the Spearman rank correlation coefficient as brain network features. The channel features extracted after filtering are mapped and stacked, converted into three-dimensional features, and then input into VSCMs of different kernel sizes to obtain feature information of different scales. Compared with the one-dimensional arrangement of channels in the traditional technology, the present invention better considers the actual distribution of electrode channels in the human brain space, and solves the problem that electrode channels that are originally adjacent in the human brain space cannot be used normally during the convolution process. At the same time, the present invention uses an upsampling layer to fuse point-by-point convolutions of different levels, and uses a convolution layer to perform a series of convolutions on the fused features and then superimposes them to obtain more comprehensive multi-scale features. The method is suitable for situations with a small number of channels and improves the fatigue driving detection rate.

Description

Technical Field

[0001] The present invention belongs to the technical field of fatigue driving detection, and in particular relates to a fatigue driving detection method based on a dual-source multi-scale feature fusion network. Background Art

[0002] Driving fatigue often affects a driver's perception, judgment, and decision-making abilities, reducing physical alertness and prolonging reaction time. Driving fatigue generally increases the probability of traffic accidents and is a major cause of them, accounting for 20% of traffic accidents and 40% of serious accidents. Therefore, reliable fatigue detection methods are crucial for driving safety. Three main types of data are used in fatigue detection: vehicle behavior characteristics, video, and physiological signals. Compared to the other two types of data, physiological signals are less susceptible to uncertain external factors. Among various physiological signals, electroencephalogram (EEG) signals are considered the gold standard for detecting brain activity and are widely used to detect driver fatigue. However, due to its nonlinearity and high sensitivity, analyzing and utilizing EEG remains a significant challenge.

[0003] Several recent studies have explored extracting multi-scale information from different features. Researchers have extracted differential entropy (DE) from each EEG signal channel as a basic feature. They then construct a feature matrix using biharmonic spline interpolation and employ a feature pyramid network (FPN) to improve the performance of electroencephalogram (EEG) emotion recognition. Some researchers have constructed four EEG feature matrices by using preprocessed signals, differential entropy, symmetric differences, and symmetric quotients to fuse different features for more comprehensive feature information. Others have used azimuthal equidistant projection (AEP) to generate two-dimensional EEGs and extract the mean power, variance power, and standard deviation power of the a, b, and g frequency bands as feature data for EEG feature maps. Starting from raw data, some methods have proposed an ERP-WTC analysis method for EEG signal channel screening and designed an AM-PCNet network to extract time, frequency, and mixed domain information from the fused spectrum of EEG and surface electromyography signals. Other methods have extracted differential entropy (DE), power spectral density (PSD), nonlinear energy (NE), and fractal dimension (FD) to classify EEG emotions using a deep classification module. In addition, some have proposed methods that use stacked multi-scale convolutional neural networks (MS-CNNs) to extract multi-scale spatiotemporal features of EEG signals using spatially distributed raw EEG signals as input. It can be seen that most of the proposed methods extract multi-scale information from raw data or a few channel features, and do not fully utilize EEG information from multiple sources.

[0004] In addition, most of the existing methods for extracting multi-scale information use a feature pyramid structure. For example, SleepFC proposed by some researchers uses a convolutional feature pyramid network. Some methods use SleepyCo-backbone as the backbone network of the feature pyramid to consider multiple feature sequences at different time and frequency scales. Another method proposes a new EEG signal denoising model DTP-Net, which consists of a densely connected time pyramid (dtp) located at two learnable time-frequency transforms. There are even more methods that directly use feature pyramids to extract multi-scale features. However, this hierarchical increasing structure is designed for the image field and cannot adapt well to the characteristics of EEG data. It is not suitable for EEG data with a small number of channels. The original feature pyramid structure, in which the convolution kernel size increases with the number of layers, is designed for the image field and cannot adapt well to the characteristics of EEG data. Summary of the Invention

[0005] The purpose of the present invention is to provide a fatigue driving detection method based on a dual-source multi-scale feature fusion network.

[0006] The present invention provides a fatigue driving detection method based on a dual-source multi-scale feature fusion network, which includes the following steps:

[0007] Step 1: Obtain EEG signals from different subjects to construct a dataset; classify fatigue levels into different categories and label the dataset;

[0008] Step 2: Filter the EEG signals in the data set at different frequency bands; calculate the differential entropy and Spearman rank correlation coefficient of the filtered EEG signals respectively, and obtain channel features based on the differential entropy and Spearman rank correlation coefficient and brain-brain network characteristics Add to the dataset;

[0009] Step 3: Construct a dual-source multi-scale feature fusion network; the dual-source multi-scale feature fusion network includes a brain-brain network feature extraction module, a channel feature extraction module and a fusion module; the inputs of the brain-brain network feature extraction module and the channel feature extraction module are the brain-brain network feature extraction module and the channel feature extraction module, respectively. and channel characteristics The brain network feature extraction module and the channel feature extraction module both include multiple branches with the same structure. Each branch includes a variable kernel size convolution module, a point-by-point convolution, and a convolution layer connected in sequence. The branches in the brain network feature extraction module and the channel feature extraction module are sorted from small to large according to the size of the convolution kernel in the variable kernel size convolution module. Point-by-point convolution is used to reduce the dimensionality of the input feature map and output the feature map. ;in, ; A is the number of branches; except for the A-th branch, the convolution layer on each branch receives the feature map And the upsampled feature map As input; the convolution layer on the last branch A only receives the feature map The brain network feature extraction module and the channel feature extraction module fuse multiple feature maps output by different branches through the brain network feature fusion module and the channel feature fusion module respectively; the fusion module is used to fuse the outputs of the brain network feature fusion module and the channel feature extraction module to obtain the prediction results;

[0010] Step 4: Use the dataset with category labels to train the dual-source multi-scale feature fusion network;

[0011] Step 5: Collect the subject's EEG signal, perform frequency division filtering on it, extract features, and input it into the trained dual-source multi-scale feature fusion network to detect the subject's fatigue level.

[0012] As a preference, in the step 2, obtaining channel characteristics and brain network characteristics The method is as follows:

[0013] Mapping differential entropy one-to-one to Two-dimensional matrix, the number of rows of the two-dimensional matrix and number of columns Varies with the mapping method and stacked in frequency band order Channel characteristics of size ; Stack the Spearman rank correlation coefficient in frequency band order to obtain Brain network characteristics of size ;in, is the number of frequency bands; c is the number of channels of the EEG signal.

[0014] Preferably, in the step three, the variable kernel size convolution module includes two layers of convolutional networks connected in sequence, and each layer of the convolutional network includes a convolutional layer, a batch normalization layer and a ReLU activation function connected in sequence.

[0015] Preferably, in step 1, the eyelid closure duration percentage index is used to represent the subject's fatigue level, and the subject's fatigue level is divided into three categories according to the index thresholds of 0.35 and 0.7.

[0016] Preferably, in the step three, the brain network feature fusion module includes a flattening layer, a discarding layer and a fully connected layer connected in series; the flattening layer is used to convert multiple feature maps output by the convolutional layer into a vector; the discarding layer is used to reduce the risk of overfitting to improve the generalization ability of the model.

[0017] Preferably, in the step three, the channel feature fusion module includes a flattening layer and a fully connected layer connected in series; the flattening layer is used to convert multiple feature maps output by the convolutional layer into a vector.

[0018] Preferably, in the step three, the fusion module includes a fully connected layer and a softmax function connected in sequence; the input of the fully connected layer is the feature vector obtained by splicing the outputs of the brain network feature fusion module and the channel feature extraction module; the softmax function is used to predict the probability that the sample belongs to each category.

[0019] As an advantage, in the step 3, the number of branches in the channel feature extraction module and the brain network feature extraction module are respectively determined by the channel feature. and brain network characteristics increases with the increase of the two-dimensional feature size.

[0020] Preferably, in step three, upsampling is performed using a nearest neighbor interpolation method.

[0021] Preferably, in step 4, a cross entropy loss function is used to guide the training, and the expression of the loss function Loss is:

[0022]

[0023] Among them, cross_entropy is the cross entropy function; α is the weight parameter; W is the parameter trained in the network; and y j are the predicted probability and the true probability respectively; j=1,2,...,N; N is the number of categories of fatigue degree; ||·||2 is the symbol of L2 norm.

[0024] The present invention has the following beneficial effects:

[0025] 1. The present invention uses an upsampling layer to fuse point-by-point convolutions of different levels, and uses a convolution layer to perform a series of convolutions on the fused features and then superimposes them to obtain more comprehensive multi-scale features. Compared with traditional fusion methods, the present invention more effectively realizes the fusion of feature information of different scales, is suitable for situations with a small number of channels, and improves the fatigue driving detection rate; at the same time, the present invention extracts channel features and brain network features through DE differential entropy and Spearman rank correlation coefficient respectively, and uses a brain network feature extraction module and a channel feature extraction module to extract brain network features and channel features respectively, effectively utilizing feature information of different sources and scales.

[0026] 2. The present invention maps and stacks the channel features extracted after frequency division filtering, converts them into three-dimensional features, and then inputs VSCMs of different kernel sizes to obtain feature information of different scales. Compared with the one-dimensional arrangement of channels in traditional technology, the present invention better considers the actual distribution of electrode channels in the human brain space, and solves the problem that electrode channels that are originally adjacent in the human brain space cannot be used normally during the convolution process. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of EEG signal frequency division feature extraction in the present invention;

[0028] Figure 2 Schematic diagram of the channel feature mapping method in the present invention.

[0029] Figure 3 Schematic diagram of the structure of the dual-source multi-scale feature fusion network in the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below with reference to the accompanying drawings.

[0031] like Figure 1 As shown, a fatigue driving detection method based on a dual-source multi-scale feature fusion network includes the following steps:

[0032] Step 1: Obtain and preprocess the EEG dataset. The EEG dataset includes the SEED-VIG dataset and the Cui dataset. The SEED-VIG dataset includes 23 subjects, with an EEG sampling frequency of 1000 Hz, which is reduced to 200 Hz after preprocessing. The number of EEG signal channels is 17. The PERCLOS (percentage of duration of eyelid closure) metric is used to represent the subject's fatigue level, ranging from 0 to 1, with higher values ​​indicating greater fatigue. Based on PERCLOS thresholds of 0.35 and 0.7, the subject's fatigue level is categorized into three categories: awake (awake < 0.35), tired (0.35 ≤ tired < 0.7), and drowsy (drowsy ≥ 0.7). Every 8 seconds of EEG signal is considered a sample, and each sample is labeled with the fatigue level category.

[0033] The Cui dataset includes 2,022 EEG samples from 11 subjects. Each EEG sample is 3 seconds long and the sampling rate is downsampled from the original 500 Hz to 128 Hz. The number of EEG electrode channels is 30. Reaction time is used to measure whether the subject is in a state of fatigue. The fatigue level of the subjects is divided into two categories: fatigue and alertness. The Cui dataset is labeled with each category, with each category containing 1,011 samples.

[0034] Step 2: Frequency division feature extraction

[0035] like Figure 1 As shown, the EEG signals in the data set are filtered at different frequency bands to obtain the filtered EEG signals. ; Where T is the number of sample points; c is the number of channels of the EEG signal; and f is the number of frequency bands. The SEED-VIG dataset was filtered into five frequency bands: δ (1-3 Hz), θ (4-7 Hz), α (8-13 Hz), β (14-30 Hz), and γ (31-50 Hz). The Cui dataset was filtered into four frequency bands: δ (1-4 Hz), θ (4-8 Hz), α (8-13 Hz), and β (13-30 Hz). In order to simultaneously utilize and compare multiple types of features in the EEG data, the DE differential entropy and Spearman rank correlation coefficient were calculated for the filtered EEG data to obtain two-dimensional features of channel features (DE differential entropy) and brain network features (Spearman rank correlation coefficient). The DE differential entropy and Spearman rank correlation coefficient The calculation formula is:

[0036]

[0037] in, is the probability density function of channel x.

[0038] When the channel x obeys a Gaussian distribution, the probability density function The expression is:

[0039]

[0040] Where σ is the standard deviation.

[0041] If the lengths of two consecutive channels x and y are both n, then the Spearman rank correlation coefficient The calculation formula is:

[0042]

[0043] in, ; is the EEG signal of channel x at the sample point at time i; y i is the EEG data of channel y at the sample point at time i; ; is the length of the channel.

[0044] Since the extracted point feature data is arranged in one-dimensional order according to the channel sequence without considering the actual distribution of the electrode channels in the human brain space, for the convolutional network, the electrode channels that are adjacent in the human brain space may be very far apart in the one-dimensional arrangement, which makes the electrode channels that are originally adjacent in the human brain space unable to be used normally during the convolution process. Therefore, for the channel feature, the actual distribution of the channel in the human brain space is used to map the one-dimensional point feature into a two-dimensional matrix, such as Figure 2 As shown; by this method, the length One-to-one mapping of one-dimensional point features to In a two-dimensional matrix of size, the empty positions are filled with 0, and the number of rows in the two-dimensional matrix and number of columns These two-dimensional matrices are stacked in order of frequency bands to obtain All 3D features constitute T channel features. For brain network features, each Spearman rank correlation coefficient is The values ​​in these adjacency matrices represent the weights of the connections between EEG channels. These adjacency matrices are stacked in the order of frequency bands to obtain All 3D features constitute T brain network features. .

[0045] Step 3: Build a dual-source multi-scale feature fusion network

[0046] like Figure 3 As shown in the figure, the dual-source multi-scale feature fusion network includes a brain network feature extraction module, a channel feature extraction module and a fusion module. The brain network feature extraction module includes four branches and a brain network feature fusion module that fuses the four branches; the four branches have the same structure, including VSCM modules (variable kernel size convolution modules), point-by-point convolution and convolution layers connected in sequence; the VSCM module is an important component of the dual-source multi-scale feature fusion network, which is used to extract brain network features. Perform feature extraction; according to the size of the convolution kernel in the VSCM module, the four branches are divided into the first branch, the second branch, the third branch and the fourth branch in descending order, and the convolution kernel sizes are 1×1, 2×2, 3×3, and 4×4 respectively; the VSCM module includes two layers of convolutional networks connected in sequence, and each layer of convolutional network includes a convolutional layer, a batch normalization layer and a ReLU activation function connected in sequence; the number of convolution kernels in the convolution layers of the first and second layers of convolutional networks are 32 and 256 respectively; the convolution layer is used to extract feature maps of different dimensions; since the convolution layer is followed by the batch normalization layer, no bias is used here; the batch normalization layer is used to reduce the gradient disappearance and gradient explosion problems and accelerate the convergence speed of the network; the ReLU activation function is used to add nonlinear factors to improve the nonlinear expression ability of the neural network; the feature map output by the first layer of convolutional network on branch a for:

[0047]

[0048] in, (·) is the convolution operation; (·) is the batch normalization operation; (·) is the ReLU activation function; is the size of the convolution kernel, k∈{1,2,3,4}.

[0049] Feature map of the second convolutional network output on branch a for:

[0050]

[0051] The convolution kernel size of point-by-point convolution is 1×1, and the number of convolution kernels is 64; point-by-point convolution is used to reduce the dimension of the feature map output by the VSCM module to reduce computational complexity and maintain network depth; the feature map output by point-by-point convolution for:

[0052]

[0053] The convolution layer on each branch a (a≠A) receives the feature map And the feature map after upsampling ; The convolution layer on the last branch A only receives feature maps ; Input of the convolutional layer for:

[0054]

[0055] in, is the upsampling operation.

[0056] The feature maps of two different scales are initially fused together through the convolution layer; the number of convolution kernels in the four convolution layers is 256, and the convolution kernel size is 3×3.

[0057] The brain network feature fusion module includes a flattening layer, a dropout layer, and a fully connected layer. The flattening layer is used to convert multiple feature maps output by the convolutional layer into a vector; the dropout layer is used to reduce the risk of overfitting to improve the generalization ability of the model; the output length of the fully connected layer is 100. The feature vector finally output by the brain network feature fusion module is for:

[0058]

[0059] in, (·) is the flattening layer; (·) is the dropout layer; (·) is a fully connected layer.

[0060] Since the channel feature map is small in size after being converted into a two-dimensional matrix, the channel feature extraction module only includes three branches and a channel feature fusion module that fuses the three branches. The branch structure of the channel feature extraction module is the same as that of the brain network feature extraction module. The channel feature fusion module includes a flattening layer and a fully connected layer. The flattening layer is used to convert the multiple feature maps output by the convolution layer into a vector. The output length of the fully connected layer is 100. The feature vector finally output by the channel feature extraction module is for:

[0061]

[0062] The fusion module includes a fully connected layer and a softmax function connected in sequence; the input of the fully connected layer is the feature vector and The output length of the concatenated feature vector of the fully connected layer is the number of categories. The softmax function is used to predict the probability that the sample belongs to each category, and the maximum value of the predicted probability is used as the prediction result y ̂.

[0063] Step 4: Training and evaluating dual-source multi-scale feature fusion network

[0064] The dual-source multi-scale feature fusion network is trained and tested using the SEED-VIG dataset and the Cui dataset respectively. The cross-entropy loss function is used to guide the training, and L2 regularization is used to make the weight parameters smoother to alleviate the overfitting problem. The expression of the loss function Loss is:

[0065]

[0066] in, is the cross entropy function; is the weight parameter; W is the parameter trained in the network; and are the true probability and the predicted probability respectively; is the L2 norm symbol; ; N is the number of categories for fatigue levels.

[0067] The initial learning rate was uniformly set to 0.001, the training batch size was 16, the epochs were set to 50, and the L2 regularization coefficient was set to 0.01 and 0.0001 on the SEED-VIG and Cui datasets, respectively. The dropout parameter was set to 0.5 in all random dropout layers in the model. The networks constructed in this paper and existing networks were tested using the SEED-VIG and Cui datasets, respectively.

[0068] Each subject in the SEED-VIG dataset has 885 samples. A five-fold cross-validation method was used independently on each subject to divide the dataset into five parts. Each part with 177 samples was used as the test set in turn, and the other samples were used as the training set. Finally, the results of the 23 subjects were summarized as the test results of the dataset. The test results of each network are shown in Table 1.

[0069] Table 1 Comparison with the current state-of-the-art methods on the SEED-VIG dataset (%)

[0070]

[0071] Because the Cui dataset contains only a small number of samples per subject, a five-fold cross-validation experiment was conducted on all 2022 samples from all subjects. This experiment divided the dataset into five parts, with each part containing 404 samples alternately serving as the test set and the remaining samples as the training set. The results of these five rounds of experiments were summarized as the test results for the dataset. The test results for each network are shown in Table 2.

[0072] Table 2 Comparison with the current state-of-the-art methods on the Cui dataset (%)

[0073]

[0074] As shown in Tables 1 and 2, the accuracy of the proposed method on the SEED-VIG dataset is 96.29%, and on the Cui dataset it is 92.83%. Compared to traditional machine learning methods (XGBoost, SVM, and ACF-SSMM), the proposed method exhibits significant advantages, achieving improvements of 13.45%, 9.19%, and 6.45%, respectively, on the SEED-VIG dataset. Furthermore, compared to graph convolutional network methods (AMCNN-DGCN, RGNN, AMD-GCN, PGCNN, and DGCNN), the proposed method better integrates functional network and feature channel information, resulting in increases of 35.47%, 9.76%, 6.35%, 5.15%, and 1.52%, respectively, on the SEED-VIG dataset, and 32.65%, 15.88%, 14.69%, and 4.85%, respectively, on the Cui dataset. Furthermore, this method demonstrates significant improvements over conventional convolutional network approaches (FBN-CNN and PSD-CNN), achieving accuracy increases of 31.97% and 4.98%, respectively, on the SEED-VIG dataset. Overall, this method is effective and surpasses current state-of-the-art methods in accuracy.

[0075] Step 5: After preprocessing the EEG signal to be detected, the preprocessed EEG signal is frequency-filtered and features are extracted; the extracted features are input into the trained dual-source multi-scale feature fusion network model to generate a detection result for the subject's fatigue level.

Claims

1. A fatigue driving detection method based on a dual-source multi-scale feature fusion network, characterized by: The following steps are involved: Step 1: Obtain EEG signals from different subjects to construct a dataset; classify fatigue levels into different categories and label the dataset; Step 2: Filter the EEG signals in the data set at different frequency bands; calculate the differential entropy and Spearman rank correlation coefficient of the filtered EEG signals respectively, and obtain channel features based on the differential entropy and Spearman rank correlation coefficient and brain network characteristics Add to the dataset; Step 3: Construct a dual-source multi-scale feature fusion network. The dual-source multi-scale feature fusion network includes a brain network feature extraction module, a channel feature extraction module, and a fusion module. The inputs of the brain network feature extraction module and the channel feature extraction module are the brain network feature extraction module and the channel feature extraction module, respectively. and channel characteristics The brain network feature extraction module and the channel feature extraction module both include multiple branches with the same structure. Each branch includes a variable kernel size convolution module, a point-by-point convolution, and a convolution layer connected in sequence. The branches in the brain network feature extraction module and the channel feature extraction module are sorted from small to large according to the size of the convolution kernel in the variable kernel size convolution module. Point-by-point convolution is used to reduce the dimensionality of the input feature map and output the feature map. ;in, ; A is the number of branches; Except for the A-th branch, the convolutional layer on each branch receives the feature map And the upsampled feature map As input; the convolution layer on the last branch A only receives the feature map The brain network feature extraction module and the channel feature extraction module fuse multiple feature maps output by different branches through the brain network feature fusion module and the channel feature fusion module respectively; the fusion module is used to fuse the outputs of the brain network feature fusion module and the channel feature extraction module to obtain the prediction results; Step 4: Use the dataset with category labels to train the dual-source multi-scale feature fusion network; Step 5: Collect the subject's EEG signal, perform frequency division filtering on it, extract features, and input it into the trained dual-source multi-scale feature fusion network to detect the subject's fatigue level.

2. The fatigue driving detection method based on a dual-source multi-scale feature fusion network according to claim 1 is characterized by: In the step 2, the channel characteristics are obtained and brain network characteristics The method is as follows: Mapping differential entropy one-to-one to Two-dimensional matrix, the number of rows of the two-dimensional matrix and number of columns Varies with the mapping method and is stacked in frequency band order. Channel characteristics of size ;Will The Spearman rank correlation coefficient of size is stacked in the order of frequency bands to obtain Brain network characteristics of size ;in, is the number of frequency bands; c is the number of channels of the EEG signal.

3. The fatigue driving detection method based on a dual-source multi-scale feature fusion network according to claim 1 is characterized by: In the step 3, the variable kernel size convolution module includes two layers of convolutional networks connected in sequence, and each layer of the convolutional network includes a convolutional layer, a batch normalization layer and a ReLU activation function connected in sequence.

4. The fatigue driving detection method based on a dual-source multi-scale feature fusion network according to claim 1 is characterized by: In the step 1, the eyelid closure duration percentage index is used to represent the subject's fatigue level, and the subject's fatigue level is divided into three categories according to the index thresholds of 0.35 and 0.

7.

5. The fatigue driving detection method based on a dual-source multi-scale feature fusion network according to claim 1 is characterized by: In the step three, the brain network feature fusion module includes a flattening layer, a dropout layer, and a fully connected layer connected in series; the flattening layer is used to convert multiple feature maps output by the convolutional layer into a vector; the dropout layer is used to reduce the risk of overfitting to improve the generalization ability of the model.

6. The fatigue driving detection method based on a dual-source multi-scale feature fusion network according to claim 1 is characterized by: In the step 3, the channel feature fusion module includes a flattening layer and a fully connected layer connected in series; the flattening layer is used to convert multiple feature maps output by the convolutional layer into a vector.

7. The method for detecting fatigue driving based on a dual-source multi-scale feature fusion network according to claim 1, characterized in that: In step three, the fusion module includes a fully connected layer and a softmax function connected in sequence; the input of the fully connected layer is the feature vector obtained by splicing the outputs of the brain network feature fusion module and the channel feature extraction module; the softmax function is used to predict the probability that the sample belongs to each category.

8. The method for detecting fatigue driving based on a dual-source multi-scale feature fusion network according to claim 1, characterized in that: In the step 3, the number of branches in the channel feature extraction module and the brain network feature extraction module are respectively and brain network characteristics increases with the increase of the two-dimensional feature size.

9. The method for detecting fatigue driving based on a dual-source multi-scale feature fusion network according to claim 1, characterized in that: In the step three, upsampling is performed using the nearest neighbor interpolation method.

10. The method for detecting fatigue driving based on a dual-source multi-scale feature fusion network according to claim 1, characterized in that: In step 4, the cross entropy loss function is used to guide the training. The expression of the loss function Loss is: in, is the cross entropy function; is the weight parameter; W is the parameter trained in the network; and are the predicted probability and the true probability respectively; ; N is the number of categories of fatigue level; is the L2 norm symbol.

Images