Driver alertness prediction method based on electroencephalogram signals and related device

Abstract

The invention discloses a driver alertness prediction method based on electroencephalogram signals and a related device, and relates to the technical field of alertness prediction.The method comprises the steps that a time-space cross multi-head attention module is designed, and the time-space cross multi-head attention module comprises an Input Embedding module, an Inception module, a first summation module, a second summation module and a time-space fusion module; after electroencephalogram signal data in the driving process of a driver is obtained, feature extraction is conducted on the electroencephalogram signal data, electroencephalogram signal features are obtained, the electroencephalogram signal features comprise space-time fusion features, and the space-time fusion features are features obtained by conducting feature extraction on the electroencephalogram signal data through a space-time cross multi-head attention module; subsequently, the electroencephalogram signal features serve as input, and the alertness of the driver in the driving process is determined through the trained alertness prediction model. The method can improve the accuracy of predicting the alertness of the driver.

Description

Technical Field

[0001] The present application relates to the technical field of alertness prediction, and particularly to a method and related device for predicting driver alertness based on electroencephalogram signals. Background Art

[0002] In the modern transportation system, the alertness of drivers plays a decisive role in road traffic safety. With the continuous increase in the number of motor vehicles, the incidence of traffic accidents remains high, and the decrease in driver alertness caused by fatigue, distraction, etc. is one of the key factors leading to traffic accidents, bringing huge losses to life and property of society. At present, the assessment of driver alertness mainly relies on subjective judgment, such as observing the driver's behavior, for example, whether yawning frequently or the eyes are blurred, etc., but such methods have great limitations, lacking objectivity and being difficult to achieve real-time and accurate assessment.

[0003] With the rapid development of technology, objective assessment techniques based on biological signal analysis have gradually emerged. As a non-invasive technique that can record brain electrical activities in real time, electroencephalogram (EEG) has gradually received attention and shown great potential in the field of driver alertness prediction. EEG signals can reflect the state of neuronal activities in the brain in real time. When the alertness of a driver changes, the neuronal activities in the brain will change accordingly, and specific characteristics will be manifested in the EEG signals. Therefore, classifying and predicting driver alertness based on EEG signals is expected to play an important role in actual traffic scenarios and effectively reduce the risk of traffic accidents caused by driver alertness problems.

[0004] In recent years, computer technology has achieved leapfrog development. In this context, deep learning methods represented by convolutional neural networks (CNNs) and Transformers, with their powerful capabilities in feature extraction, have brought new breakthroughs to the processing methods of EEG signals. EEG signals contain rich time-domain, frequency-domain, and spatial features. However, in past studies, when using traditional machine learning models to carry out driver alertness prediction work, the prediction accuracy was low. Summary of the Invention

[0005] The purpose of the present application is to provide a method and related device for predicting driver alertness based on electroencephalogram signals, which can improve the prediction accuracy of driver alertness.

[0006] To achieve the above purpose, the present application provides the following solutions:

[0007] First aspect, the present application provides a method for predicting driver alertness based on electroencephalogram (EEG) signals. The method for predicting driver alertness based on EEG signals includes:

[0008] Obtain EEG signal data during the driver's driving process;

[0009] Extract features from the EEG signal data to obtain EEG signal features. The EEG signal features include spatio-temporal fusion features. The spatio-temporal fusion features are features obtained by using a spatio-temporal cross multi-head attention module to extract features from the EEG signal data. The spatio-temporal cross multi-head attention module includes an Input Embedding module, an Inception module, a first summation module, a second summation module, and a spatio-temporal fusion module. The input end of the Input Embedding module is used to input the EEG signal data. The output end of the Input Embedding module is respectively connected to the input end of the Inception module and the first input end of the first summation module. The second input end of the first summation module is used to input the position encoding of the EEG signal data. The output end of the first summation module is connected to the first input end of the spatio-temporal fusion module. The output end of the Inception module is connected to the first input end of the second summation module. The second input end of the second summation module is used to input the position encoding of the EEG signal data. The output end of the second summation module is connected to the second input end of the spatio-temporal fusion module. The spatio-temporal fusion module is used to process the query vector output by the first summation module and the key vector and value vector output by the second summation module, and output the spatio-temporal fusion features;

[0010] Using the EEG signal features as input, determine the alertness of the driver during the driving process by using a trained alertness prediction model.

[0011] Second aspect, the present application provides a device for predicting driver alertness based on EEG signals. The device for predicting driver alertness based on EEG signals includes:

[0012] A data acquisition module for obtaining EEG signal data during the driver's driving process;

[0013] A feature extraction module is used to extract features from the electroencephalogram (EEG) signal data to obtain EEG signal features. The EEG signal features include spatio-temporal fusion features, which are features obtained by using a spatio-temporal cross multi-head attention module to extract features from the EEG signal data. The spatio-temporal cross multi-head attention module includes an InputEmbedding module, an Inception module, a first summation module, a second summation module, and a spatio-temporal fusion module. The input end of the InputEmbedding module is used to input the EEG signal data. The output end of the Input Embedding module is respectively connected to the input end of the Inception module and the first input end of the first summation module. The second input end of the first summation module is used to input the position encoding of the EEG signal data. The output end of the first summation module is connected to the first input end of the spatio-temporal fusion module. The output end of the Inception module is connected to the first input end of the second summation module. The second input end of the second summation module is used to input the position encoding of the EEG signal data. The output end of the second summation module is connected to the second input end of the spatio-temporal fusion module. The spatio-temporal fusion module is used to process the query vector output by the first summation module and the key vector and value vector output by the second summation module, and output the spatio-temporal fusion feature;

[0014] An alertness prediction module is used to use the EEG signal features as input and determine the alertness of the driver during driving by using a trained alertness prediction model.

[0015] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the computer program to implement the above-mentioned driver alertness prediction method based on EEG signals.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned driver alertness prediction method based on EEG signals is implemented.

[0017] In a fifth aspect, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the above-mentioned driver alertness prediction method based on EEG signals is implemented.

[0018] According to the specific embodiments provided by the present application, the present application has the following technical effects:

[0019] The present application provides a method for predicting driver alertness based on electroencephalogram (EEG) signals and related devices. The designed spatio-temporal cross multi-head attention module includes an Input Embedding module, an Inception module, a first summation module, a second summation module, and a spatio-temporal fusion module. After obtaining the EEG signal data during the driver's driving process, the EEG signal data is subjected to feature extraction to obtain EEG signal features, where the EEG signal features include spatio-temporal fusion features. The spatio-temporal fusion features are the features obtained by using the spatio-temporal cross multi-head attention module to extract features from the EEG signal data. Subsequently, using the EEG signal features as input, the trained alertness prediction model is used to determine the alertness of the driver during the driving process. By designing the spatio-temporal cross multi-head attention module, the present application can extract the spatio-temporal fusion features of the EEG signal data, and then predict the alertness of the driver during the driving process based on the EEG signal features including the spatio-temporal fusion features, which can make full use of the time features and space features to predict the driver's alertness and improve the prediction accuracy of the driver's alertness. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 It is an application environment diagram of a method for predicting driver alertness based on EEG signals provided in Embodiment 1 of the present application.

[0022] Figure 2 It is a schematic flowchart of a method for predicting driver alertness based on EEG signals provided in Embodiment 1 of the present application.

[0023] Figure 3 It is a schematic structural diagram of the spatio-temporal cross multi-head attention module provided in Embodiment 1 of the present application.

[0024] Figure 4 It is a schematic structural diagram of the multi-head attention layer provided in Embodiment 1 of the present application.

[0025] Figure 5 It is a schematic technical route diagram of a method for predicting driver alertness based on EEG signals provided in Embodiment 1 of the present application.

[0026] Figure 6 It is a schematic microstate analysis flowchart provided in Embodiment 1 of the present application.

[0027] Figure 7Schematic diagram of functional modules of a driver alertness prediction device provided in Embodiment 2 of this application.

[0028] Figure 8 Schematic diagram of the structure of a computer device provided in Embodiment 3 of this application. Detailed implementation manners

[0029] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0030] Embodiment 1

[0031] The driver alertness prediction method based on electroencephalogram (EEG) signals provided in the embodiments of this application can be applied to an application environment as shown in Figure 1 . Among them, the terminal communicates with the server through the network. The data storage system can store the data that the server needs to process. The data storage system can be set up separately, integrated on the server, or placed on the cloud or other servers. The terminal can send the EEG signal data to be processed to the server. After receiving the EEG signal data to be processed, for the EEG signal data to be processed, the server extracts features from the EEG signal data to obtain EEG signal features, and the EEG signal features include spatio-temporal fusion features; using the EEG signal features as input, the trained alertness prediction model is used to determine the alertness of the driver during the driving process. The server can feedback the prediction result of the alertness for the EEG signal data obtained to the terminal.

[0032] In addition, in some embodiments, the driver alertness prediction method based on EEG signals can also be implemented by the server or the terminal alone. For example, the terminal can directly process the EEG signal data to be processed, or the server can obtain the EEG signal data to be processed from the data storage system and process the EEG signal data to be processed.

[0033] Among them, the terminal can be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, Internet of Things (IoT) devices, and portable wearable devices. The IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers, and can also be a cloud server.

[0034] In an exemplary embodiment, as shown in Figure 2As shown, a method for predicting driver alertness based on electroencephalogram (EEG) signals is provided. This method is executed by a computer device, which can be specifically executed by a computer device such as a terminal or a server alone, or jointly executed by a terminal and a server. In the embodiments of the present application, taking the application of this method to Figure 1 the server in

[0035] Step S1: Obtain the EEG signal data during the driver's driving process.

[0036] Step S2: Extract features from the EEG signal data to obtain EEG signal features. The EEG signal features include spatio-temporal fusion features. The spatio-temporal fusion features are features obtained by using a spatio-temporal cross multi-head attention module to extract features from the EEG signal data. The spatio-temporal cross multi-head attention module includes an Input Embedding module, an Inception module, a first summation module, a second summation module, and a spatio-temporal fusion module. The input end of the Input Embedding module is used to input the EEG signal data. The output end of the Input Embedding module is respectively connected to the input end of the Inception module and the first input end of the first summation module. The second input end of the first summation module is used to input the position encoding of the EEG signal data. The output end of the first summation module is connected to the first input end of the spatio-temporal fusion module. The output end of the Inception module is connected to the first input end of the second summation module. The second input end of the second summation module is used to input the position encoding of the EEG signal data. The output end of the second summation module is connected to the second input end of the spatio-temporal fusion module. The spatio-temporal fusion module is used to process the query vector output by the first summation module and the key vector and value vector output by the second summation module, and output the spatio-temporal fusion features.

[0037] Step S3: Use the trained alertness prediction model to determine the alertness of the driver during the driving process with the EEG signal features as the input.

[0038] By implementing the above steps S1 to S3, in this embodiment, starting from the spatio-temporal dimension, comprehensively considering the time features and space features of the EEG signal, spatio-temporal fusion features are obtained. Subsequently, based on the EEG signal features including spatio-temporal fusion features, the alertness of the driver during the driving process is predicted, which can improve the prediction accuracy of the driver's alertness.

[0039] At present, certain achievements have been made in the research on the classification and prediction of driver alertness based on EEG signals. Deep learning models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and their variants have gradually been applied in this field, demonstrating powerful feature learning and classification capabilities. However, current research often only focuses on time-domain features and frequency-domain features, while ignoring spatial features and not conducting targeted analysis and utilization of them, resulting in the following problems:

[0040] (1) EEG signals are usually recorded using multiple electrodes placed at different positions on the scalp surface. Each electrode can independently record the electrical activities in its nearby area. Due to the spatial distance between the electrodes, the brain electrical activity information captured by them differs both in time and space. These differences reflect the temporal and spatial characteristics of EEG signals. Therefore, how to adjust and optimize the model structure while paying attention to both temporal and spatial characteristics is an issue that requires further research.

[0041] (2) The signal propagation between neurons can occur between different regions of the brain, forming neural pathways of different distances. Therefore, EEG signals have spatial characteristics at different scales, and it is difficult to completely capture these multi-scale information through a single convolutional neural network.

[0042] In view of this, this embodiment innovatively designs a brand-new model. Starting from the spatio-temporal dimension, this brand-new model comprehensively considers the temporal and spatial characteristics of EEG signals and aims to achieve more accurate classification and prediction of driver alertness. Specifically, to solve the problem of being unable to simultaneously obtain the temporal and spatial characteristics of EEG signals and completely capture multi-scale information, this embodiment constructs a brand-new model that fuses temporal and spatial characteristics based on the Transformer model and proposes a Time-Space Multi-Head Cross-Attention (TS-MHA) module.

[0043] In this embodiment, the Inception module is introduced at the input end of the Transformer model. The spatial and temporal features of the extracted EEG signals are cross-fused through the Transformer model to extract spatio-temporal fusion characteristics, that is, the fusion of temporal and spatial features is achieved based on the time-space multi-head cross-attention module, as Figure 3 shown. This time-space multi-head cross-attention module includes an Input Embedding module, an Inception module, a first summation module, a second summation module, and a spatio-temporal fusion module. The following is a detailed introduction to each part:

[0044] (1) Input Embedding (Input Embedding) module

[0045] The input end of the Input Embedding module is used to input EEG signal data. The output end of the Input Embedding module is respectively connected to the input end of the Inception module and the first input end of the first summation module. What the EEG signal obtains through the Input Embedding module is a low-dimensional, continuous and information-rich vector representation, which can capture the key features in the EEG signal (it can be set to perform time-domain feature extraction) and remove noise. The Input Embedding module is used to output time features.

[0046] (2) First summation module

[0047] The first input end of the first summation module is connected to the output end of the Input Embedding module. The second input end of the first summation module is used to input the position encoding of the EEG signal data. The output end of the first summation module is connected to the first input end of the spatio-temporal fusion module. The first summation module includes an addition layer. The first summation module is used to add the position encoding element by element to the time features output by the Input Embedding module and output a query vector.

[0048] The calculation formula of the query vector is:

[0049]

[0050] Among them, is the query vector; is the time feature output by the Input Embedding module; PE is the position encoding of the EEG signal data; W Q is the first weight matrix.

[0051] (3) Inception module

[0052] The input end of the Inception module is connected to the output end of the Input Embedding module. The output end of the Inception module is connected to the first input end of the second summation module. The Inception module is used to perform multi-scale feature extraction on the time features output by the Input Embedding module, obtain multi-scale spatial features, and further perform channel splicing to output the first spatial feature and the second spatial feature.

[0053] The Inception module can adopt any existing Inception module. As an example, the Inception module in this embodiment includes a first branch, a second branch, a third branch, and a filter concatenation layer. The input ends of the first branch, the second branch, and the third branch are the input ends of the Inception module, all of which are connected to the output end of the InputEmbedding module. The output ends of the first branch, the second branch, and the third branch are all connected to the input end of the filter concatenation layer. The output end of the filter concatenation layer is the output end of the Inception module. Among them, the first branch includes a 1×1 convolutional layer, the second branch includes a 1×1 convolutional layer and a 3×3 convolutional layer connected in sequence, and the third branch includes a 1×1 convolutional layer, a 3×3 convolutional layer, and a 3×3 convolutional layer connected in sequence. Different branches extract spatial features of different scales, and then perform channel concatenation through the filter concatenation layer. The finally output is a concatenated spatial feature. Since the subsequent Transformer model requires the division of three types of vectors, this concatenated spatial feature is used as the first spatial feature and the second spatial feature That is, the first spatial feature and the second spatial feature are equal.

[0054] (4) The second summation module

[0055] The first input end of the second summation module is connected to the output end of the Inception module. The second input end of the second summation module is used to input the position encoding of the electroencephalogram signal data. The output end of the second summation module is connected to the second input end of the spatio-temporal fusion module. The second summation module includes an addition layer. The second summation module is used to add the position encoding element by element to the first spatial feature and the second spatial feature output by the Inception module respectively, and output the key vector and the value vector.

[0056] The calculation formulas for the key vector and the value vector are as follows:

[0057]

[0058]

[0059] Among them, is the key vector; is the first spatial feature output by the Inception module; PE is the position encoding of the electroencephalogram signal data; W K is the second weight matrix; is the value vector; is the second spatial feature output by the Inception module; W Vis the third weight matrix.

[0060] (5) Spatiotemporal Fusion Module

[0061] The first input end of the spatiotemporal fusion module is connected to the output end of the first summation module, and the second input end of the spatiotemporal fusion module is connected to the output end of the second summation module. The spatiotemporal fusion module is used to process the query vector output by the first summation module and the key vector and value vector output by the second summation module, and output spatiotemporal fusion features.

[0062] As an example, the spatiotemporal fusion module of this embodiment includes a cross-attention unit and a linear layer connected in sequence. The cross-attention unit includes a multi-head attention layer, a first residual connection layer (Add), a first normalization layer (Norm), a feed-forward fully connected layer, a second residual connection layer (Add), and a second normalization layer (Norm). The first input end of the multi-head attention layer is connected to the output end of the first summation module, the second input end of the multi-head attention layer is connected to the output end of the second summation module, the output end of the multi-head attention layer is connected to the first input end of the first residual connection layer, the second input end of the first residual connection layer is connected to the output end of the first summation module, the output end of the first residual connection layer is connected to the input end of the first normalization layer, the output end of the first normalization layer is connected to the input end of the feed-forward fully connected layer and the first input end of the second residual connection layer, the output end of the feed-forward fully connected layer is connected to the second input end of the second residual connection layer, the output end of the second residual connection layer is connected to the input end of the second normalization layer, the output end of the second normalization layer is the output end of the cross-attention unit, the output end of the cross-attention unit is connected to the input end of the linear layer, and the output end of the linear layer is the output end of the spatiotemporal cross multi-head attention module, which is used to output spatiotemporal fusion features.

[0063] In the alertness classification prediction task, since the distribution of EEG signals on the scalp is uneven, the projection positions of the electrical activities generated in different brain regions on the scalp are different. At the same time, EEG signals change dynamically over time, and they will change rapidly with factors such as the brain's activity state and external stimuli. This unique temporal and spatial characteristic is often ignored. To solve this problem, this embodiment introduces a spatiotemporal cross multi-head attention module to capture long-term dependence relationships and temporal and spatial features. TS-MHA combines the multi-head attention mechanism (Multi-HeadAttention) and spatiotemporal analysis, and enhances the signal modeling ability by utilizing the information in the temporal and spatial domains.

[0064] The processing flow of TS-MHA is as follows: The received EEG signal passes through the Input Embedding module. Its main function is to map the original EEG signal to a feature space that is more suitable for the model to process and learn, enabling the model to better capture the patterns and relationships in the data, improving the performance and generalization ability of the model. The EEG signal obtains temporal features via the Input Embedding module.

[0065] Then, two-way processing is carried out. In the first branch, the temporal features respectively pass through three branches of the Inception module: 1×1 convolution, 1×1 convolution + 3×3 convolution, 1×1 convolution + 3×3 convolution + 3×3 convolution. The 1×1 convolution is used to perform convolution operations on the channel dimension to extract local spatial features between channels, and the other two branches are used to extract multi-scale spatial features to capture the global spatial features between channels. The Filter Concat operation concatenates the feature maps extracted by these different-scale convolutional kernels in the channel dimension, thereby fusing multi-scale features together, enabling the model to utilize information at different scales simultaneously and capture multi-scale spatial characteristics. After passing through the Inception module, multi-scale spatial features (i.e., the first spatial feature and the second spatial feature ) are extracted. In the second branch, the temporal features will directly add positional encoding information for subsequent processing. The query vector is obtained by multiplying the temporal features extracted after passing through the Input Embedding module, adding the positional encoding PE, and then multiplying by the first weight matrix W Q . The key vector and the value vector are obtained by multiplying the multi-scale spatial features extracted after passing through the Inception module, adding the positional encoding PE, and then multiplying by the second weight matrix W K and the third weight matrix W V , that is, multiplying the input vector by three different weight matrices to map the query vector, key vector, and value vector.

[0066] After that, as Figure 4 shown, the multi-head attention layer consists of H Scaled Dot-Product Attention layers. H can be 8. Scaled Dot-Product Attention is one of the core components in the Transformer model. It is an attention mechanism that allows the model to jointly focus on different information from different positions, dynamically aggregates the information of the values by calculating the similarity between the query and the key, thereby helping the model better understand the input sequence. The output features of the multi-head attention layer The calculation formula is as follows:

[0067]

[0068] Among them, the output vector of the H-th head W O is the weight matrix.

[0069] When calculating the output vector of a single head, the calculated will be connected to the Scaled Dot-ProductAttention layer shown on the right after passing through the linear layer respectively, and then divided by the scalar factor Figure 4 and the softmax function is used to obtain the weights of these values. The calculation formula is as follows:

[0070]

[0071] where d k is the dimension of the key vector used to calculate the dot product of the query and the key and is part of the scaling factor to ensure numerical stability.

[0072] Finally, the output vector that fuses the spatio-temporal features obtained from the multi-head attention layer is input into the residual connection (Add) and layer normalization (Norm) to solve the problem of gradient disappearance and improve the generalization ability of the model. After passing through the feed-forward fully connected layer for dimensional transformation and non-linear transformation, it is then input into the residual connection (Add) and layer normalization (Norm) to solve the problem of gradient disappearance and improve the generalization ability of the model. Finally, the result is input into a linear layer for classification to obtain the spatio-temporal fusion features.

[0073] In this embodiment, the linear layer is the fully connected layer.

[0074] Based on this, in this embodiment, the features extracted by the classical method and the features extracted by the deep learning model Inception-Transformer (i.e., the spatio-temporal cross-attention module) are fused and then unified for vigilance prediction, as Figure 5 shown, including the following steps:

[0075] (1) Perform data preprocessing on the collected EEG signals.

[0076] The collected EEG signals are preprocessed using the EEGLAB toolbox in MATLAB software. First, downsampling is performed to reduce the EEG signals from 1000 Hz to 500 Hz, which can reduce the space occupied by the data. Then, a band-pass filter of 1 - 30 Hz is applied to the EEG signals. At the same time, to remove interference such as power frequency noise, a notch point is set at 50 Hz for notch filtering of the EEG signals. The main purpose is to remove noise and interference signals and enhance the EEG activities in the target frequency band. Finally, artifacts are removed. EEG signals are easily affected by various artifacts during the acquisition process. Artifacts are mainly divided into physiological artifacts (such as eye movements, blinks, and myoelectric activities) and non-physiological artifacts (such as power frequency interference and poor electrode contact). These artifacts will mask or distort the true EEG activities, affecting the analysis and interpretation of EEG signals. Therefore, removing artifacts is a key step in EEG signal processing. By methods such as Independent Component Analysis (ICA) or wavelet transform, artifacts can be effectively separated and removed, thereby improving the signal quality and ensuring the accuracy of subsequent feature extraction and classification. Removing artifacts not only helps improve the reliability of EEG signals but also provides a cleaner data basis for alertness prediction.

[0077] In related research, multi-channel EEG signal data is usually spliced in a simple cascaded manner, that is, each sample is represented as a one-dimensional vector whose length is equal to the number of time points multiplied by the number of channels. However, this representation of one-dimensional vectors can only reflect the information of a single channel and cannot effectively capture the mutual relationships and influences between different channels. Therefore, this embodiment proposes to use a two-dimensional matrix to splice multi-channel EEG signal data. The preprocessed EEG signals are segmented. The EEG signals are divided into multiple 10s samples, and then the EEG signals are set in the matrix form of a CNN (i.e., channels × sample points) as follows:

[0078]

[0079] where X is the EEG signal matrix; is the EEG signal of the b-th segment of the a-th electrode channel. The b-th segment of the EEG signal can be 10 s. a = 1, 2,..., j, where j is the number of electrode channels of the EEG signal, and b = 1, 2,..., i, where i is the number of segments of the EEG signal, that is, the number of samples.

[0080] At this time, in this embodiment, the EEG signal data during the driver's driving process is obtained. The EEG signal data includes the EEG signals collected by each electrode channel, and feature extraction is performed on the EEG signal data to obtain EEG signal features.

[0081] Among them, feature extraction is performed on the electroencephalogram (EEG) signal data to obtain EEG signal features, which specifically include: segmenting the EEG signals of each electrode channel in the EEG signal data respectively to obtain an EEG signal matrix, where the element in the ath row and bth column of the EEG signal matrix is the bth segment of the EEG signal of the ath electrode channel; performing feature extraction on the EEG signal matrix to obtain EEG signal features.

[0082] (2) Input the EEG signal matrix into two feature extraction methods respectively for feature extraction.

[0083] This embodiment presents two methods, classical feature extraction and Inception-Transformer feature extraction. Below, methods for extracting different features are given. The specific features are obtained by selecting from all available features according to the purpose of alertness prediction.

[0084] The classical feature extraction method is as follows:

[0085] (1.1) Calculate time-domain features.

[0086] This embodiment extracts time-domain features such as mean and variance.

[0087] Mean: The mean represents the average value of the EEG signal within a certain time window, reflecting the central position of the EEG signal. Its calculation formula is:

[0088]

[0089] where μ is the mean; N is the total number of time points within the time window; x i is the EEG signal value at the ith time point.

[0090] Variance: The variance represents the degree of dispersion of the EEG signal within a certain time window, reflecting the volatility of the EEG signal. Its calculation formula is:

[0091]

[0092] where σ 2 is the variance.

[0093] (1.2) Calculate frequency-domain features.

[0094] Based on the influence of alertness on EEG rhythm waves, key features such as the Power Ratio Index (PRI), Relative Intensity Ratio (RIR), and Maximum-to-Mean Power Ratio (mmrPS) are manually selected as frequency-domain features.

[0095] Power Spectral Density (PSD): The power spectral density represents the power distribution of EEG signals at different frequencies. It describes the power magnitude of each frequency component in the EEG signal and intuitively shows the distribution characteristics of the energy of the EEG signal on the frequency axis. Calculate the energy proportion of the Delta (1 - 4 Hz), Theta (4 - 8 Hz), Alpha (8 - 13 Hz), and Beta (13 - 30 Hz) frequency bands.

[0096] Power index ratio: The power index ratio is obtained by calculating the ratio between the power spectral density values in different frequency ranges. The power ratio of the Delta - Theta frequency band (1 - 8 Hz) and the Alpha - Beta frequency band (8 - 30 Hz) is used to measure the change in low - frequency power. Its calculation formula is:

[0097]

[0098] where PRI is the power index ratio; e J1 is the first lower limit, specifically 1 Hz; e J2 is the first upper limit, specifically 8 Hz; O y (e) is the signal power; e i1 is the second lower limit, specifically 8 Hz; e i2 is the second upper limit, specifically 30 Hz.

[0099] Relative intensity ratio: Obtained by comparing the power of a certain frequency band with the total power. Its calculation formula is:

[0100]

[0101] where RIR is the relative intensity ratio; e j1 is the third lower limit; e j2 is the third upper limit; E is the full frequency band, specifically 1 - 30 Hz.

[0102] In this embodiment, the relative intensity ratio will be calculated for four frequency bands respectively. At this time, for the Delta (1 - 4 Hz) frequency band, the third lower limit is specifically 1 Hz and the third upper limit is specifically 4 Hz. For the Theta (4 - 8 Hz) frequency band, the third lower limit is specifically 4 Hz and the third upper limit is specifically 8 Hz. For the Alpha (8 - 13 Hz) frequency band, the third lower limit is specifically 8 Hz and the third upper limit is specifically 13 Hz. For the Beta (13 - 30 Hz) frequency band, the third lower limit is specifically 13 Hz and the third upper limit is specifically 30 Hz.

[0103] Maximum average power ratio: The ratio of the maximum power of a certain frequency band to the average value of the total power, which can evaluate the relative stability and change degree of the EEG frequency band. Its calculation formula is:

[0104]

[0105] Among them, mmrPS is the maximum average power ratio.

[0106] (1.3) Calculate the non-linear features.

[0107] This embodiment extracts non-linear features such as approximate entropy (ApEn) and sample entropy (SampEn).

[0108] Approximate entropy: Approximate entropy is an index to measure the regularity of a time series, used to evaluate the degree of repetition of patterns in a signal. The smaller the value, the more regular the signal; the larger the value, the more complex the signal. The calculation process is as follows:

[0109] For a given time series x(n), where n = 1, 2, …, N (N is the number of time points), first reconstruct it into a set of m-dimensional vectors X(i):

[0110] x(i) = [x(i), x(i + 1), …, x(i + m - 1)], where i = 1, 2, …, N - m + 1;

[0111] Among them, m is a constant.

[0112] Define the distance d[X(i), X(j)] between two vectors X(i) and X(j) as the maximum value of the absolute value of the difference of the corresponding elements, that is:

[0113]

[0114] For a given threshold r, define as the ratio of the number of j (j = 1, 2, …, N - m + 1 and j ≠ i) that satisfy d[X(i), X(j)] ≤ r to N - m, that is:

[0115]

[0116] Calculate Then increase the vector dimension to m + 1, and repeat the above steps to obtain C m+1 (r).

[0117] The calculation formula of approximate entropy is:

[0118]

[0119] Among them, ApEn(m, r, N) is the approximate entropy.

[0120] Sample entropy: Sample entropy is an improvement over approximate entropy and is used to measure the complexity of a time series. Different from approximate entropy, sample entropy excludes self-comparison and its calculation is more stable. The calculation formula is as follows:

[0121]

[0122] where SampEn(m, r, N) is the sample entropy;

[0123] (1.4) Calculate the microstate-related time features.

[0124] EEG microstate refers to a transient state with a relatively stable spatial distribution and specific pattern presented by multi-channel EEG signals in an extremely short time during the process of brain nerve activity. EEG microstate believes that the electrical activity of the brain is not continuous and stable, but can be decomposed into a series of relatively stable microstates that quickly switch in time. Each microstate corresponds to a specific pattern of brain nerve activity, reflecting the overall functional state and information processing method of the brain at a certain moment. In this embodiment, by performing microstate analysis on the driver's EEG signals and extracting the corresponding time features, the microstate analysis process is as Figure 6 shown. First, calculate the GFP (Global Field Power) value of the EEG signal. The electrophysiological data synchronously recorded at multiple scalp points usually presents as a series of equipotential maps, which show the spatial distribution of brain electrical activity at different time points. Usually, the GFP curve is used to describe brain electrical activity.

[0125] The calculation formula of GFP at time t is:

[0126]

[0127] where L is the total number of electrodes; u i is the voltage value corresponding to electrode i at time t; is the average value of the voltage values corresponding to all electrodes at time t.

[0128] Then, extract the EEG topographic map at the randomly selected GFP peak, and use the improved K-means clustering algorithm to perform clustering analysis on the EEG topographic map to obtain the optimal clustering. To determine the number of optimal clusters (i.e., microstates), two fitting metric parameters, GEV (Global explained variance) and CV (The cross-validation criterion), need to be considered. GEV is a metric used to evaluate the similarity between each EEG sample and its assigned microstate. The calculation formula is:

[0129]

[0130] where x n is the nth EEG sample; α ln is the microstate assigned to the nth EEG sample; GFP n is the global electric field power of the nth EEG sample; S is the total number of EEG samples; GFP n' is the global electric field power of the n'th EEG sample; is the sum of the squares of GFP for S EEG samples, which serves as a normalization factor for normalizing the squared value of Corr(x n , α ln )·GFP n .

[0131] CV is usually related to the residual noise, and its calculation formula is:

[0132]

[0133] where is the estimated value of the residual noise variance; C is the number of EEG channels; K is the number of clusters, i.e., the number of microstates.

[0134] When selecting the number of microstates, it is usually desired to obtain a higher GEV value and a lower CV value.

[0135] After determining the number of microstates, the microstates are backfitted to all EEG samples. Meanwhile, to improve the fitting quality, the microstate sequence of the EEG samples is smoothed in time, and microstates with a duration shorter than 30 ms are excluded. After completing the microstate analysis, the following temporal features are extracted from the microstate topographies:

[0136] 1) Duration: It represents the average duration of each microstate without interruption, reflecting the stability of the underlying neural components.

[0137] 2) Occurrence frequency, also known as incidence rate: It represents the frequency of each microstate occurring within a one-second interval, reflecting the trend of activation of the underlying neural generators.

[0138] 3) Temporal coverage: It represents the proportion of time occupied by each microstate in the entire recording, which is calculated from the duration and incidence rate.

[0139] 4) Transition probability: It represents the likelihood of transitioning from one microstate to another, describing the dynamic characteristics of the brain switching between different functional states and reflecting the flexibility and adaptability of brain function.

[0140] The Inception-Transformer feature extraction method is as follows:

[0141] To take into account the temporal and spatial characteristics of EEG signals, as well as complex global spatial relationships, an Inception module is introduced at the input end of the Transformer model to capture multi-scale features, and a spatio-temporal fusion feature is specifically extracted using a spatio-temporal cross-attention module.

[0142] In this embodiment, the EEG signal matrix X is subjected to feature extraction according to the above process. To enrich different types of features, at this time, in this embodiment, the EEG signal features include spatio-temporal fusion features, temporal domain features, frequency domain features, non-linear features, and microstate-related time features. The temporal domain features include mean and variance. The frequency domain features include power index ratio, relative intensity ratio, and maximum average power ratio. The non-linear features include approximate entropy and sample entropy. The microstate-related time features include duration, occurrence frequency, time coverage rate, and transition probability.

[0143] Based on this, this embodiment completes the acquisition of EEG data fusion features.

[0144] (3) After the preprocessed EEG signals are respectively subjected to feature extraction by classical methods and Inception-Transformer, the extracted features are subjected to standardization, feature dimensionality reduction, and feature alignment processing. The processed features are input into a fully connected layer for feature combination, and classification prediction is performed through a Sigmoid activation function to obtain the alertness based on the EEG signal.

[0145] At this time, in this embodiment, using the EEG signal features as input, the trained alertness prediction model is used to determine the alertness of the driver during the driving process.

[0146] Among them, using the EEG signal features as input and using the trained alertness prediction model to determine the alertness of the driver during the driving process specifically includes: performing standardization processing, feature dimensionality reduction, and feature alignment on the EEG signal features to obtain processed features; using the processed features as input and using the trained alertness prediction model to determine the alertness of the driver during the driving process, and the alertness includes low alertness and high alertness; among them, the trained alertness prediction model includes a fully connected layer and a Sigmoid activation function layer connected in sequence.

[0147] During the training process of the trained alertness prediction model, in this embodiment, the dataset is divided into a training set and a test set. The dataset includes samples and labels. The samples are sample electroencephalogram signal features, and the labels are the corresponding sample alertness of the sample electroencephalogram signal features. The sample alertness is set according to user requirements. Specifically, the dataset is divided into a training set and a test set according to a ratio of 8:2. The training set is used for model training and parameter optimization, and the test set is used to evaluate the generalization performance and final effect of the model. Through this division method, it can ensure that the model fully learns the features of the data during the training process, and at the same time verifies its performance in actual applications on an independent test set, thereby avoiding the overfitting problem and improving the reliability and robustness of the model. In addition, the division of the training set and the test set strictly follows the principle of random assignment to ensure the balance of data distribution and further guarantee the scientificity and repeatability of the experimental results.

[0148] In this embodiment, the results are evaluated through three indicators: accuracy, F1 score, and area under the ROC curve (AUC-ROC). Specifically, the performance evaluation of this embodiment aims to verify the accuracy and generalization ability of the proposed method. In addition to using common classification indicators such as accuracy, the F1 score and the area under the ROC curve indicators are also introduced to comprehensively evaluate the classification performance of the model from multiple dimensions. The specific definitions of the evaluation indicators are as follows:

[0149] Accuracy: Accuracy is the most intuitive indicator in the classification task, indicating the proportion of the number of samples correctly predicted by the model to the total number of samples. Its calculation formula is:

[0150]

[0151] Among them, Acc is the accuracy; TP (True Positive) is the number of samples correctly predicted as the positive class by the model; TN (True Negative) is the number of samples correctly predicted as the negative class by the model; FP (False Positive) is the number of samples wrongly predicted as the positive class by the model; FN (False Negative) is the number of samples wrongly predicted as the negative class by the model.

[0152] AUC-ROC: AUC-ROC is the area under the ROC curve, with a value range of [0, 1], which is used to measure the performance of the model at different classification thresholds. The closer it is to 1, the better the model performance.

[0153] The F1 score is the harmonic mean of precision and recall, which is used to comprehensively evaluate the classification performance of the model. The higher the F1 score, the better the classification performance of the model. Its calculation formula is:

[0154]

[0155] Currently, the alertness classification prediction models based on EEG signals mainly rely on convolutional neural networks (CNNs) and long short-term memory networks (LSTMs). CNNs gradually extract the features of EEG signals through multiple convolutional and pooling operations. EEG signals have high-dimensional and complex spatial distribution characteristics, and the spatial relationships (such as distance and topology) between EEG electrodes are very important for signal analysis. However, traditional CNNs mainly rely on convolutional kernels to extract local features and may not be able to fully capture the complex global spatial relationships in EEG signals. At the same time, EEG signals are typical time series data with strong temporal dependence, but CNNs have weak temporal modeling capabilities and are difficult to capture long-term dependence relationships. Although LSTMs have strong temporal modeling capabilities when processing EEG signals, their modeling capabilities for spatial features and long sequence data are limited. To better capture long-term dependence relationships, a more efficient Transformer model is adopted, which overcomes the problems of gradient disappearance or gradient explosion when traditional models process long sequences. This embodiment has the following advantages: It is proposed to use the Inception module as the spatial feature extraction module of the Transformer model, and input the temporal features and spatial features of EEG signals into the Transformer model for fusion, which can well capture global dependence relationships; A spatio-temporal cross multi-head attention module is proposed. In this module, the query vector is obtained by superimposing the temporal features extracted after passing through the Input Embedding module, adding positional encoding, and then multiplying by the weight matrix W Q The key vector and the value vector are obtained by superimposing the multi-scale spatial features extracted after passing through the Inception module, adding positional encoding, and then multiplying by the weight matrix W K and W V The query vector the key vector and the value vector are used as the inputs of the spatio-temporal cross multi-head attention module for cross fusion to obtain spatio-temporal fusion features; It is proposed to use a two-dimensional matrix to realize the splicing of multi-channel EEG data, and set the EEG signal as the input in the matrix form of CNN. As a two-dimensional structure, the matrix can store and express more structural information. It can simultaneously capture the relationship change rules of multi-channels in the time domain and spatial domain, making the analysis of multi-channel EEG information more comprehensive.

[0156] At present, the classification and prediction of alertness based on EEG signals represented by deep learning have been widely studied. The more commonly used network structures are CNN and RNN, and the representative model is C-LSTM. Although time and space feature extraction has been carried out, the research on the attention mechanism is lacking. Based on the Transformer model, this embodiment combines Inception and spatio-temporal cross multi-head attention mechanism, effectively improving the performance of alertness classification prediction. The specific advantages are as follows:

[0157] (1) Deep learning models for the classification and prediction of alertness based on EEG signals usually have two network structures: convolutional neural network and recurrent neural network. The convolutional neural network slides the convolutional kernel on the EEG data and extracts local spatial features in the data through convolutional operations. The recurrent neural network divides the EEG data into sequences of multiple time steps according to the time order. For each time step in the input sequence, the recurrent neural network takes the input of the current time step and the hidden state of the previous time step as inputs and calculates through specific computing units, which can effectively capture the long-term dependencies in the EEG signals and better process EEG data with time series characteristics. However, since EEG signals have both time and space characteristics, single convolutional neural networks and recurrent neural networks ignore the spatio-temporal characteristics of EEG signals. To extract the time and space features of EEG signals, the encoder of the Transformer is improved, and a spatio-temporal cross self-attention mechanism is proposed, which fuses the time features and space features from EEG signals and further learns global spatio-temporal features.

[0158] (2) The signal transmission between neurons is carried out through chemical synapses and electrical synapses. When signals propagate between neural structures at different scales, they are affected by various factors such as the length of nerve fibers, conduction speed, and synaptic strength. These factors result in different ways of propagation and integration of neural information at different spatial scales, thus making EEG signals exhibit multi-scale spatial characteristics. Traditional convolutional neural networks can only extract single-scale characteristics. Therefore, the Inception module is used for the extraction of multi-scale spatial features of EEG signals. By using parallel convolutional layers and pooling operations with different filter sizes (such as 1x1, 3x3, 5x5), multi-scale spatial features can be efficiently captured.

[0159] (3) Each channel in the EEG multi-channel signals has spatial correlation, and the signals collected by electrodes in different regions will affect each other. The two-dimensional matrix can directly represent the corresponding relationship between different channels at different time points through rows and columns, so that cross-channel spatio-temporal features can be extracted, better reflecting the correlation information of EEG signals between brain regions.

[0160] The present application also provides an application scenario, which applies the above-mentioned driver alertness prediction method based on electroencephalogram (EEG) signals. Specifically, the driver alertness prediction method based on EEG signals provided in this embodiment can be applied in a warning scenario, which includes a prediction stage and a warning stage. The prediction stage is used to predict the driver's alertness during driving, and the alertness includes low alertness and high alertness. The warning stage is used to emit a sound to remind the driver when the alertness is low alertness, so as to achieve the purpose of safe driving. The driver alertness prediction method based on EEG signals provided in this embodiment belongs to the prediction stage.

[0161] Embodiment 2

[0162] Based on the same inventive concept, the embodiment of the present application also provides a driver alertness prediction device based on EEG signals for implementing the above-mentioned driver alertness prediction method based on EEG signals. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the driver alertness prediction device based on EEG signals provided below can refer to the limitations on the driver alertness prediction method based on EEG signals in the above text, and will not be repeated here.

[0163] In an exemplary embodiment, as Figure 7 shown, a driver alertness prediction device based on EEG signals is provided. The driver alertness prediction device based on EEG signals includes:

[0164] A data acquisition module M1, configured to acquire EEG signal data of the driver during driving.

[0165] A feature extraction module M2 is configured to extract features from the electroencephalogram signal data to obtain electroencephalogram signal features. The electroencephalogram signal features include spatio-temporal fusion features, which are features obtained by using a spatio-temporal cross multi-head attention module to extract features from the electroencephalogram signal data. The spatio-temporal cross multi-head attention module includes an InputEmbedding module, an Inception module, a first summation module, a second summation module, and a spatio-temporal fusion module. The input end of the InputEmbedding module is configured to input the electroencephalogram signal data. The output end of the InputEmbedding module is respectively connected to the input end of the Inception module and the first input end of the first summation module. The second input end of the first summation module is configured to input the position encoding of the electroencephalogram signal data. The output end of the first summation module is connected to the first input end of the spatio-temporal fusion module. The output end of the Inception module is connected to the first input end of the second summation module. The second input end of the second summation module is configured to input the position encoding of the electroencephalogram signal data. The output end of the second summation module is connected to the second input end of the spatio-temporal fusion module. The spatio-temporal fusion module is configured to process the query vector output by the first summation module and the key vector and value vector output by the second summation module, and output the spatio-temporal fusion feature.

[0166] An alertness prediction module M3 is configured to use the electroencephalogram signal features as input and determine the alertness of a driver during driving by using a trained alertness prediction model.

[0167] Embodiment 3

[0168] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal, and its internal structure diagram may be as shown in Figure 8 The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method for predicting a driver's alertness based on electroencephalogram signals.

[0169] Those skilled in the art can understand that Figure 8 the structure shown in Figure 8 is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0170] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the driver alertness prediction method based on electroencephalogram signals in Embodiment 1 is implemented.

[0171] Embodiment 4

[0172] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the driver alertness prediction method based on electroencephalogram signals in Embodiment 1 is implemented.

[0173] Embodiment 5

[0174] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the driver alertness prediction method based on electroencephalogram signals in Embodiment 1 is implemented.

[0175] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0176] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0177] Specific examples are used in this article to elaborate on the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, based on the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation on this application.

Claims

1. A method for predicting driver alertness based on EEG signals, characterized in that: The driver alertness prediction method based on EEG signals includes: Obtaining the driver's EEG signal data during driving; The EEG signal data is subjected to feature extraction to obtain EEG signal features; the EEG signal features include spatiotemporal fusion features, which are features obtained by extracting features from the EEG signal data using a spatiotemporal cross-multi-head attention module, the spatiotemporal cross-multi-head attention module includes an Input Embedding module, an Inception module, a first summation module, a second summation module and a spatiotemporal fusion module, the input end of the Input Embedding module is used to input the EEG signal data, the Input The output end of the Embedding module is respectively connected to the input end of the Inception module and the first input end of the first summation module, the second input end of the first summation module is used to input the position code of the EEG signal data, the output end of the first summation module is connected to the first input end of the spatiotemporal fusion module, the output end of the Inception module is connected to the first input end of the second summation module, the second input end of the second summation module is used to input the position code of the EEG signal data, the output end of the second summation module is connected to the second input end of the spatiotemporal fusion module, and the spatiotemporal fusion module is used to process the query vector output by the first summation module and the key vector and value vector output by the second summation module, and output the spatiotemporal fusion feature; The EEG signal features are used as input and the trained alertness prediction model is used to determine the driver's alertness during driving.

2. The method for predicting driver alertness based on EEG signals according to claim 1, characterized in that: Extracting features from the EEG signal data to obtain EEG signal features specifically includes: The EEG signal of each electrode channel in the EEG signal data is processed in segments to obtain an EEG signal matrix; the element in the a-th row and the b-th column in the EEG signal matrix is ​​the b-th segment of the EEG signal of the a-th electrode channel; Feature extraction is performed on the EEG signal matrix to obtain EEG signal features.

3. The method for predicting driver alertness based on EEG signals according to claim 1, characterized in that: The EEG signal features also include time domain features, frequency domain features, nonlinear features and microstate-related time features. The time domain features include mean and variance, the frequency domain features include power index ratio, relative intensity ratio and maximum average power ratio, the nonlinear features include approximate entropy and sample entropy, and the microstate-related time features include duration, occurrence frequency, time coverage and transition probability.

4. The method for predicting driver alertness based on EEG signals according to claim 1, characterized in that: The Inception module includes a first branch, a second branch, a third branch and a filter splicing layer, the input end of the first branch, the input end of the second branch and the input end of the third branch are all connected to the output end of the Input Embedding module, the output end of the first branch, the output end of the second branch and the output end of the third branch are all connected to the input end of the filter splicing layer, and the output end of the filter splicing layer is the output end of the Inception module; The first branch includes a 1×1 convolutional layer; the second branch includes a 1×1 convolutional layer and a 3×3 convolutional layer connected in sequence; the third branch includes a 1×1 convolutional layer, a 3×3 convolutional layer and a 3×3 convolutional layer connected in sequence; The spatiotemporal fusion module includes a cross-attention unit and a linear layer connected in sequence. The cross-attention unit includes a multi-head attention layer, a first residual connection layer, a first normalization layer, a feedforward fully connected layer, a second residual connection layer and a second normalization layer. The first input end of the multi-head attention layer is connected to the output end of the first summation module, the second input end of the multi-head attention layer is connected to the output end of the second summation module, the output end of the multi-head attention layer is connected to the first input end of the first residual connection layer, the second input end of the first residual connection layer is connected to the output end of the first summation module, the output end of the first residual connection layer is connected to the input end of the first normalization layer, the output end of the first normalization layer is connected to the input end of the feedforward fully connected layer and the first input end of the second residual connection layer, the output end of the feedforward fully connected layer is connected to the second input end of the second residual connection layer, the output end of the second residual connection layer is connected to the input end of the second normalization layer, and the output end of the second normalization layer is the output end of the cross-attention unit.

5. The method for predicting driver alertness based on EEG signals according to claim 1, characterized in that: The calculation formulas of the query vector, the key vector and the value vector are: in, is the query vector; is the temporal feature output by the Input Embedding module; PE is the position encoding of the EEG signal data; W Q is the first weight matrix; is the key vector; is the first spatial feature output by the Inception module; W K is the second weight matrix; is a value vector; is the second spatial feature output by the Inception module; W V is the third weight matrix.

6. The method for predicting driver alertness based on EEG signals according to claim 1, characterized in that: Taking the EEG signal features as input, the alertness of the driver during driving is determined using the trained alertness prediction model, specifically including: Performing standardization, feature dimension reduction and feature alignment on the EEG signal features to obtain processed features; The processed features are used as input and a trained alertness prediction model is used to determine the driver's alertness during driving; wherein the trained alertness prediction model includes a fully connected layer and a Sigmoid activation function layer connected in sequence.

7. A driver alertness prediction device based on EEG signals, characterized in that: The driver alertness prediction device based on EEG signals comprises: A data acquisition module, used to acquire the driver's EEG signal data during driving; A feature extraction module is used to extract features from the EEG signal data to obtain EEG signal features; the EEG signal features include spatiotemporal fusion features, which are features obtained by extracting features from the EEG signal data using a spatiotemporal cross-multi-head attention module. The spatiotemporal cross-multi-head attention module includes an InputEmbedding module, an Inception module, a first summation module, a second summation module and a spatiotemporal fusion module. The input end of the InputEmbedding module is used to input the EEG signal data. The output end of the Embedding module is respectively connected to the input end of the Inception module and the first input end of the first summation module, the second input end of the first summation module is used to input the position code of the EEG signal data, the output end of the first summation module is connected to the first input end of the spatiotemporal fusion module, the output end of the Inception module is connected to the first input end of the second summation module, the second input end of the second summation module is used to input the position code of the EEG signal data, the output end of the second summation module is connected to the second input end of the spatiotemporal fusion module, and the spatiotemporal fusion module is used to process the query vector output by the first summation module and the key vector and value vector output by the second summation module, and output the spatiotemporal fusion feature; The alertness prediction module is used to use the EEG signal characteristics as input and use a trained alertness prediction model to determine the driver's alertness during driving.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the driver alertness prediction method based on EEG signals as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for predicting driver alertness based on electroencephalogram signals described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for predicting driver alertness based on electroencephalogram signals described in any one of claims 1 to 6 is implemented.

Images