Driver risk response detection method and device based on frequency-aware time convolution model

Abstract

This application proposes a driver risk response detection method and apparatus based on a frequency-aware temporal convolution model, belonging to the field of data processing technology. The method involves preprocessing the acquired current EEG signal by denoising, and then inputting the preprocessed EEG signal into a frequency-aware temporal convolution model. Spatial and temporal feature extraction modules are used to sequentially extract spatial and temporal features from the preprocessed EEG signal, yielding spatiotemporal features. A frequency-aware enhancement module converts the current EEG signal into a frequency domain sequence and extracts enhanced features. The spatiotemporal and enhanced features are then input into a fusion classification module to obtain the driver risk response detection result. This application proposes a frequency-aware temporal convolution model based on joint analysis of time series and frequency, and applies it to detect driver risk responsiveness, improving detection accuracy, contributing to the study of human factors in driving, and helping to reduce traffic accidents.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a driver risk response detection method and apparatus based on a frequency-aware temporal convolution model. Background Technology

[0002] Driver risk response capability refers to the timeliness with which a driver reacts to potential hazards or emergencies, significantly impacting traffic safety. Driver reaction time is crucial and is affected by various factors such as distraction and fatigue. In today's mixed traffic environment, whether a human driver directly handles unexpected events or acts as a safety supervisor for autonomous vehicles in complex situations, a high level of risk response capability is essential to ensure traffic safety. Therefore, investigating driver risk response capability is a very important issue.

[0003] When drivers are drowsy or engaged in tasks unrelated to driving, their risk reaction ability decreases, potentially making them less able to avoid emergencies. Current research in this field primarily focuses on monitoring human data to assess a driver's current risk reaction ability; however, image-based studies are affected by external factors such as lighting and occlusion. Therefore, many studies utilize physiological signals, such as EEG signals (electroencephalograms).

[0004] However, existing EEG processing and modeling methods have certain limitations. Summary of the Invention

[0005] This application provides a driver risk response detection method and apparatus based on a frequency-aware temporal convolution model to solve the problem that existing driver risk response detection methods cannot comprehensively consider time-series signals and frequency domain features, resulting in low detection accuracy.

[0006] In a first aspect of this application, a driver risk response detection method based on a frequency-aware temporal convolution model is proposed, the method comprising:

[0007] The current EEG signal acquired by the EEG signal electrode is subjected to noise reduction preprocessing to obtain the current EEG signal after noise reduction preprocessing;

[0008] The noise-reduced preprocessed current EEG signal is input into the frequency-aware temporal convolution model;

[0009] From the time domain dimension, through the spatial feature extraction module and the temporal feature extraction module of the frequency-aware temporal convolution model, the time series analysis method is used to sequentially extract spatial and temporal features from the current EEG signal after noise reduction and preprocessing to obtain spatiotemporal features.

[0010] The frequency-aware enhancement module of the frequency-aware temporal convolution model converts the current EEG signal after noise reduction preprocessing into a frequency domain sequence, performs feature enhancement from the frequency domain dimension using frequency analysis methods, and extracts the enhanced features.

[0011] The spatiotemporal features and the enhanced features are input into the fusion classification module of the frequency-aware temporal convolution model to obtain the driver risk response detection result. The driver risk response detection result is used to characterize the timeliness of the driver's response to potential dangers or emergencies during driving.

[0012] In one optional embodiment of this application, the frequency sensing enhancement module includes a first transform layer, a low-pass filter layer, a fully connected linear layer, a zero-padding layer, an inverse transform layer, and a linear layer;

[0013] The frequency-aware enhancement module of the frequency-aware temporal convolution model converts the noise-reduced preprocessed EEG signal into a frequency domain sequence, performs feature enhancement using frequency analysis methods in the frequency domain, and extracts enhanced features, including:

[0014] The input noise-reduced preprocessed current EEG signal is passed through the first transform layer to obtain the frequency domain sequence;

[0015] The frequency domain sequence is sequentially passed through the low-pass filter layer, the fully connected linear layer, and the zero-padding layer to obtain the processed frequency domain sequence.

[0016] The processed frequency domain sequence is sequentially passed through the inverse transform layer and the linear layer to obtain the enhanced feature.

[0017] In one optional embodiment of this application, the frequency-aware enhancement module of the frequency-aware temporal convolution model converts the noise-reduced preprocessed current EEG signal into a frequency domain sequence, performs feature enhancement using frequency analysis methods in the frequency domain, and extracts enhanced features, specifically including:

[0018] The input EEG signal after noise reduction preprocessing is subjected to channel normalization processing, and the normalized current EEG signal is transformed from the time domain dimension to the frequency domain dimension using the first transform layer to obtain the frequency domain sequence.

[0019] The frequency domain sequence is truncated by the low-pass filter layer to obtain filtered frequency band data.

[0020] After upsampling the filtered frequency band data using the fully connected linear layer, the high-frequency portion of the upsampled frequency band data is padded with zeros using the zero-padding layer to obtain the processed frequency domain data.

[0021] The inverse transform layer converts the processed frequency domain data from the frequency domain dimension into an EEG signal in the time domain dimension.

[0022] The linear layer is used to reduce the dimensionality of the time-domain EEG signal and extract the enhanced features.

[0023] In one optional embodiment of this application, the spatial feature extraction module includes a first 2D convolutional layer and a first batch of normalization layers. The size of the convolution kernel in the first 2D convolutional layer is set to channel*1, where channel represents the channel of the convolution kernel in the first 2D convolutional layer, so as to perform spatial convolution on signals of different channels along the time series.

[0024] Spatial feature extraction is performed on the current EEG signal after noise reduction preprocessing, including:

[0025] The current EEG signal after noise reduction preprocessing is input into the first 2D convolutional layer to perform spatial convolution and obtain the spatial convolution result;

[0026] The spatial convolution result is input into the first batch of normalization layers to obtain the spatial features.

[0027] In one optional embodiment of this application, the temporal feature extraction module is connected after the spatial feature extraction module. The temporal feature extraction module includes: a second 2D convolutional layer, a second batch normalization layer, a nonlinear activation layer, and a max pooling layer. The size of the convolutional kernel in the second 2D convolutional layer is set to 1*length, where length represents the length of the convolutional kernel in the second 2D convolutional layer.

[0028] The time feature extraction of the current EEG signal after noise reduction preprocessing includes:

[0029] The spatial features are input into the second 2D convolutional layer and temporal convolution is performed to obtain the temporal convolution result;

[0030] The temporal convolution result is input into the second batch of normalization layers to obtain the first spatiotemporal feature;

[0031] The first spatiotemporal feature is input into the nonlinear activation layer and subjected to nonlinearization processing to obtain the second spatiotemporal feature.

[0032] The second spatiotemporal feature is input into the max pooling layer for enhancement processing to obtain the spatiotemporal feature.

[0033] In one optional embodiment of this application, the current EEG signal acquired by the EEG signal electrode is subjected to noise reduction preprocessing by a 1-50Hz bandpass filter.

[0034] In one optional embodiment of this application, the frequency-aware temporal convolutional model is trained using EEG samples carrying labels for high-risk or low-risk responses as training data. The training process of the frequency-aware temporal convolutional model includes:

[0035] By inputting EEG samples carrying high-risk response or low-risk response labels into the frequency-aware temporal convolutional model to be trained, the driver risk response detection results corresponding to the EEG samples are output.

[0036] Based on the driver risk response detection results and corresponding labels of the EEG samples, calculate the cross-entropy loss;

[0037] The frequency-aware temporal convolutional model to be trained is optimized by the cross-entropy loss to obtain the frequency-aware temporal convolutional model.

[0038] A second aspect of this application provides a driver risk response detection device based on a frequency-aware temporal convolution model, the device comprising:

[0039] The preprocessing module is used to perform noise reduction preprocessing on the current EEG signal acquired by the EEG signal electrodes to obtain the noise-reduced current EEG signal;

[0040] The input module is used to input the noise-reduced preprocessed current EEG signal into the frequency-aware temporal convolution model;

[0041] The spatiotemporal feature extraction module is used to extract spatiotemporal features from the time domain dimension by using the spatial feature extraction module and the temporal feature extraction module of the frequency-aware temporal convolution model, and by employing time series analysis methods to sequentially extract spatial and temporal features from the current EEG signal after noise reduction and preprocessing, thereby obtaining spatiotemporal features.

[0042] The enhanced feature extraction module is used to convert the current EEG signal after noise reduction preprocessing into a frequency domain sequence through the frequency-aware enhancement module of the frequency-aware temporal convolution model, perform feature enhancement from the frequency domain dimension using frequency analysis methods, and extract enhanced features.

[0043] The detection result acquisition module is used to input the spatiotemporal features and the enhanced features into the fusion classification module of the frequency-aware temporal convolution model to obtain the driver risk response detection result. The driver risk response detection result is used to characterize the timeliness of the driver's response to potential dangers or emergencies during driving.

[0044] In a third aspect of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the driver risk response detection method based on the frequency-aware temporal convolution model described in any of the first aspects above.

[0045] In a fourth aspect of this application, a computer-readable storage medium is provided, on which a computer program / instruction is stored, which, when executed by a processor, implements the driver risk response detection method based on the frequency-aware temporal convolution model described in any one of the first aspects above.

[0046] This application includes the following advantages: It proposes a driver risk response detection method and apparatus based on a frequency-aware temporal convolution model. The method involves preprocessing the current EEG signal acquired by EEG signal electrodes to reduce noise, resulting in a preprocessed EEG signal. This preprocessed EEG signal is then input into a frequency-aware temporal convolution model. From a temporal perspective, the model's spatial and temporal feature extraction modules, using time series analysis, sequentially extract spatial and temporal features to obtain spatiotemporal features. The model's frequency-aware enhancement module converts the preprocessed EEG signal into a frequency domain sequence, and frequency analysis is used to enhance features and extract enhanced features. Finally, the spatiotemporal features and enhanced features are input into the model's fusion classification module to obtain the driver risk response detection result. This result characterizes the timeliness of a driver's response to potential dangers or emergencies during driving. This application proposes a frequency-aware temporal convolution model based on joint analysis of time series and frequency, and applies it to detect drivers' risk responsiveness, which improves detection accuracy, helps to study human factors in driving, and also helps to reduce traffic accidents. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating the steps of a driver risk response detection method based on a frequency-aware temporal convolution model proposed in an embodiment of this application.

[0049] Figure 2This is a flowchart illustrating a driver risk response detection method based on a frequency-aware temporal convolution model proposed in an embodiment of this application.

[0050] Figure 3 This is a schematic diagram of the network structure of a frequency-aware temporal convolutional model proposed in an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of a process for extracting enhanced features using a frequency-aware enhancement module, as proposed in an embodiment of this application.

[0052] Figure 5 This is an architecture diagram of a driver risk response detection device based on a frequency-aware temporal convolution model provided in an embodiment of this application;

[0053] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] Driver risk response capability refers to the timeliness with which a driver reacts to potential hazards or emergencies, significantly impacting traffic safety. Driver reaction time is crucial and is affected by various factors such as distraction and fatigue. In today's mixed traffic environment, whether a human driver directly handles unexpected events or acts as a safety supervisor for autonomous vehicles in complex situations, a high level of risk response capability is essential to ensure traffic safety. Therefore, investigating driver risk response capability is a very important issue.

[0056] When drivers are drowsy or engaged in non-driving related tasks (NDRTs), their risk reaction ability decreases, potentially making them less able to avoid emergencies. Current research in this field primarily focuses on monitoring human body data; for example, some studies detect driver drowsiness by analyzing the degree of eye closure and mouth opening in facial images. Additionally, some studies use half-body images to detect whether a person is engaged in NDRTs, such as using a mobile phone or adjusting a radio. These studies can assess a driver's current risk reaction ability. However, image-based research is affected by external factors such as lighting and occlusion. Therefore, many studies use physiological signals, such as EEG signals (electroencephalogram).

[0057] However, existing EEG processing and modeling methods have certain limitations.

[0058] Based on this, this application considers combining frequency awareness with spatiotemporal convolution to construct a frequency-aware temporal convolution model that integrates features from both frequency and time series analysis. This model can transform time series data into the frequency domain, where frequency analysis methods are used for data processing and feature extraction. In the time domain, time series analysis methods can be used for modeling and prediction, and this model can be applied to EEG-based driver risk responsiveness detection.

[0059] In a first aspect, this application provides a driver risk response detection method based on a frequency-aware temporal convolution model, see below. Figure 1 , Figure 1 This is a flowchart illustrating the steps of a driver risk response detection method based on a frequency-aware temporal convolution model proposed in this application. The method includes the following steps:

[0060] Step 101: Perform noise reduction preprocessing on the current EEG signal acquired by the EEG signal electrode to obtain the noise-reduced current EEG signal;

[0061] Step 102: Input the current EEG signal after noise reduction preprocessing into the frequency-aware temporal convolution model;

[0062] Step 103: From the time domain dimension, through the spatial feature extraction module and the temporal feature extraction module of the frequency-aware temporal convolution model, the time series analysis method is used to sequentially extract spatial features and temporal features from the current EEG signal after noise reduction preprocessing to obtain spatiotemporal features;

[0063] Step 104: Through the frequency-aware enhancement module of the frequency-aware temporal convolution model, the current EEG signal after noise reduction preprocessing is converted into a frequency domain sequence, and feature enhancement is performed from the frequency domain dimension using frequency analysis methods, and enhanced features are extracted;

[0064] Step 105: Input the spatiotemporal features and the enhanced features into the fusion classification module of the frequency-aware temporal convolution model to obtain the driver risk response detection result. The driver risk response detection result is used to characterize the timeliness of the driver's response to potential dangers or emergencies during driving.

[0065] To clearly explain the driver risk response detection method based on a frequency-aware temporal convolution model proposed in this application, the following will combine... Figure 2 Provided as an example, Figure 2This is a flowchart illustrating a driver risk response detection method based on a frequency-aware temporal convolution model proposed in this application.

[0066] In specific implementation step 101, the EEG signal acquisition method used in this application is as follows: An event-related lane departure paradigm is employed, with participants aged 22 to 28 years. Participants are assigned continuous driving tasks in a virtual reality simulator, and random lane departure events are introduced. Once the vehicle deviates from the center lane, participants are required to quickly return to the center lane, and EEG signals reflecting the participant's reaction time are recorded. The EEG signals are acquired using 30 EEG signal electrodes at a sampling rate of 500Hz. This application performs noise reduction preprocessing on the acquired current EEG signals to obtain a noise-reduced current EEG signal, thereby eliminating the impact of noise on the accuracy of subsequent detection results.

[0067] In one optional embodiment of this application, the acquired current EEG signal is preprocessed for noise reduction using a 1-50Hz bandpass filter. The 1-50Hz bandpass filter removes high-frequency signals above 50Hz from the current EEG signal, thus eliminating signal noise and achieving noise reduction processing of the current EEG signal.

[0068] In specific implementation step 102, the current EEG signal (corresponding to) is collected. Figure 2 After denoising the raw signal (in the image), the current EEG signal (corresponding to the denoised signal) is then processed. Figure 2 The input frequency-aware temporal convolutional model (corresponding to the Preprocessed Signal) is a Preprocessed Signal. Figure 2 The model (in the text) collects frequency and temporal features of the current EEG signal and analyzes the timeliness of the driver's risk response based on these features. The network structure of the frequency-aware temporal convolutional model described above can be found in section 3. Figure 3 This is a schematic diagram of the network structure of a frequency-aware temporal convolutional model proposed in an embodiment of this application. The frequency-aware temporal convolutional model includes: a spatial feature extraction module (corresponding to...) Figure 3 The module used for spatial feature extraction), and the temporal feature extraction module (corresponding to...) Figure 3 The module used for Temporal feature extraction), and the frequency sensing enhancement module (corresponding to...) Figure 3 The modules used in Frequency Awareness Enhancement and the Fusion Classification module (corresponding to...) Figure 3 (Modules used in Fusion and classification).

[0069] In specific implementation step 103, the acquired current EEG signal is a time-series signal. Based on this, from the time domain dimension, through the spatial feature extraction module and temporal feature extraction module of the frequency-aware temporal convolution model, a time-series analysis method is used to sequentially extract spatial and temporal features from the denoised preprocessed current EEG signal, obtaining spatiotemporal features. The aforementioned time-series analysis method refers to using the spatial feature extraction module and temporal feature extraction module of the frequency-aware temporal convolution model to extract spatial and temporal features from the EEG signal.

[0070] In one optional embodiment of this application, since EEG signals have spatial coupling, they can be considered as a mixture of signals from different brain regions. For the input current EEG signal, spatial features are first extracted using a spatial feature extraction module. This spatial feature extraction module includes a first 2D convolutional layer and a first batch of normalization layers. The size of the convolution kernel in the first 2D convolutional layer is set to channel*1, where channel represents the channel of the convolution kernel in the first 2D convolutional layer, to perform spatial convolution on signals from different channels along the time series. Specifically, spatial feature extraction of the current EEG signal after noise reduction preprocessing includes the following process: First, the current EEG signal after noise reduction preprocessing is input into the first 2D convolutional layer for spatial convolution to obtain the spatial convolution result; after spatial convolution, the spatial convolution result is input into the first batch of normalization layers for normalization processing to promote the stability and convergence speed of the frequency-aware temporal convolution model, thereby obtaining spatial features. The calculation formula for normalization processing using the first batch of normalization layers is as follows:

[0071]

[0072] In the formula, BN represents the first normalization layer, x represents the spatial convolution result, μ and σ represent the mean and standard deviation of the current EEG signal after noise reduction preprocessing, respectively, and γ and β represent the learnable scaling and offset parameters, respectively.

[0073] In one optional embodiment of this application, after spatial feature extraction is completed, convolution is applied to extract temporal features. This temporal feature extraction is achieved through a temporal feature extraction module connected after the spatial feature extraction module. The temporal feature extraction module includes: a second 2D convolutional layer, a second batch normalization layer, a nonlinear activation layer, and a max-pooling layer. The size of the convolution kernel in the second 2D convolutional layer is set to 1*length, where length represents the length of the convolution kernel in the second 2D convolutional layer. By configuring convolution kernels of specific length and appropriate stride, temporal convolution is performed to facilitate the capture of temporal patterns within the data. Specifically, temporal feature extraction is performed on the current EEG signal after the above noise reduction preprocessing, including the following process: inputting the spatial features into the second 2D convolutional layer for temporal convolution to obtain a temporal convolution result; then inputting the temporal convolution result into the second batch normalization layer for normalization processing to obtain a first spatiotemporal feature; after obtaining the first spatiotemporal feature, inputting the first spatiotemporal feature into the nonlinear activation layer (corresponding to...). Figure 3 The ReLU in the model is nonlinearized to obtain a second spatiotemporal feature, which enhances the representational ability of the frequency-aware temporal convolution model. Finally, the second spatiotemporal feature is input into the max pooling layer for enhancement to obtain the spatiotemporal feature (corresponding to...). Figure 3 Spatial-temporal feature in (the context of the problem).

[0074] In specific implementation step 104, after obtaining the spatiotemporal features, in order to enhance the features using frequency-aware technology, this application uses a frequency-aware enhancement module of a frequency-aware temporal convolution model to convert the current EEG signal after the above-mentioned noise reduction preprocessing into a frequency domain sequence, performs feature enhancement from the frequency domain dimension using frequency analysis methods, and extracts the enhanced features (corresponding to...). Figure 3 The frequency analysis methods mentioned above refer to methods such as Fourier transform (FFT), discrete Fourier transform (DFT), and real fast Fourier transform (rFFT).

[0075] In one optional embodiment of this application, see [reference] Figure 4 , Figure 4This is a flowchart illustrating a frequency-aware enhancement module for extracting enhanced features, as proposed in an embodiment of this application. The frequency-aware enhancement module includes a first transform layer, a low-pass filter layer, a fully connected linear layer, a zero-padding layer, an inverse transform layer, and a linear layer. Through the frequency-aware temporal convolution model, the frequency-aware enhancement module converts the denoised preprocessed current EEG signal into a frequency domain sequence, performs feature enhancement using frequency analysis methods in the frequency domain, and extracts the enhanced features. The process includes the following steps: First, the input time-series signal, i.e., the denoised preprocessed current EEG signal, is passed through the first transform layer to obtain a frequency domain sequence; then, the frequency domain sequence is sequentially passed through the low-pass filter layer, the fully connected linear layer, and the zero-padding layer to obtain a processed frequency domain sequence; finally, the processed frequency domain sequence is sequentially passed through the inverse transform layer and the linear layer to obtain the enhanced features.

[0076] In another optional embodiment of this application, the frequency-aware enhancement module of the frequency-aware temporal convolution model converts the noise-reduced preprocessed current EEG signal into a frequency domain sequence, performs feature enhancement using frequency analysis methods in the frequency domain, and extracts the enhanced features. Specifically, the process includes the following steps:

[0077] First, the input time series signal (corresponding to) Figure 4 The "Time Sequence" refers to the current EEG signal after noise reduction preprocessing undergoing channel normalization. The first transform layer then transforms the normalized EEG signal from the time domain to the frequency domain, resulting in a frequency domain sequence (corresponding to...). Figure 4 The transformation here is achieved through the real-fast Fourier transform (rFFT), which utilizes the symmetry of real-valued sequences in the frequency domain to facilitate the conversion of complex-valued sequences in the frequency domain into half-length real-valued sequences. Specifically, for a real-valued sequence of length N, rFFT transforms it into a complex-valued sequence of length N / 2+1, encapsulating the spectral information of the sequence.

[0078] After frequency domain transformation by the first transform layer, the resulting frequency sequence represents the signal components at different frequencies. Then, a low-pass filter layer is used to truncate the frequency band in the frequency domain sequence (corresponding to...). Figure 4 Low-pass filtering (LPF) is used to suppress sidelobe interference after filtering and eliminate interference from high-frequency signals. This can be achieved by truncating the frequency domain sequence at specific positions. This truncation operation can be understood as eliminating components in the frequency domain above the cutoff position to obtain the filtered frequency band data (corresponding to...). Figure 4 (Low-frequency in the middle).

[0079] For example, assume the desired cutoff frequency is f.cutoff Then the corresponding cutoff position k cutoff The following formula can be used for calculation:

[0080] In the formula, f s This represents the sampling rate of the frequency domain sequence.

[0081] The low-pass filter layer effectively removes frequencies above the cutoff frequency f. cutoff The frequency components are used to achieve the desired low-pass filtering effect.

[0082] Subsequently, a fully connected linear layer is used to upsample the filtered frequency band data (corresponding to...). Figure 4 In LinearUp-sampling, the upsampled frequency band data corresponds to Figure 4 In the context of Interpolated Sequences, upsampling aims to increase the sampling rate of a signal, making it more refined in the time domain. Linear interpolation is typically used to interpolate frequency domain sequences. In the frequency domain, linear interpolation usually requires interpolating adjacent points on the frequency axis.

[0083] For example, assuming we have amplitude values ​​A(f1) and A(f2) corresponding to frequencies f1 and f2, and we want to perform interpolation at a frequency f between f1 and f2, the formula for linear interpolation can be expressed as:

[0084]

[0085] In the formula, A(f) represents the interpolation result at frequency f.

[0086] After upsampling, zero-padding is applied to the high-frequency portion of the upsampled frequency band data (corresponding to...). Figure 4 Zero padding is used to ensure the consistency and integrity of the output data, resulting in processed frequency domain data (corresponding to...). Figure 4 (Padding Sequence in the middle);

[0087] For example, in the frequency domain sequence, k>k cutoff The sampling points are set to zero, that is:

[0088]

[0089] In k≤k cutoff The sampling points are not set to zero.

[0090] Subsequently, the processed frequency domain data is converted from the frequency domain dimension to a time domain dimension EEG signal (corresponding to...) through an inverse transform layer. Figure 4The Enhanced Sequence (ES) in this context refers to converting frequency domain data into time-series signals (EEG signals in the time domain). Here, the inverse transform layer is implemented using irFFT (inverse rFFT).

[0091] Finally, the time-domain EEG signal is dimensionality reduced using the linear layer (corresponding to...). Figure 4 The Dimensionality Reduction algorithm is used to extract enhanced features, which are then used for subsequent processing and analysis.

[0092] In specific implementation step 105, after obtaining the enhanced features, the spatiotemporal features and the enhanced features are input into the fusion classification module of the frequency-aware temporal convolutional model. The spatiotemporal features and the enhanced features are fused, and the fused features are classified to obtain the driver risk response detection result. This process is implemented through a fully connected layer, corresponding to... Figure 2 The Contact layer, Linear layer, ReLU layer, and Linear layer in the text.

[0093] In an optional embodiment of this application, the driver risk response detection result is used to characterize the timeliness of the driver's response to potential hazards or emergencies during driving. This driver risk response detection result includes high-risk responses (corresponding to...). Figure 2 High Risk Responsiveness and Low Risk Response (corresponding to) Figure 2 The Low Risk Responsiveness (Low Risk Response), High Risk Response, and Low Risk Response are determined as follows: Driver risk responsiveness is assessed based on driver reaction time. The local reaction time (RT) for each input EEG signal refers to the time it takes for the driver to react to a lane departure event. The global reaction time (Global RT) represents the average reaction time within a 90-second window prior to the lane departure event. The fifth percentage of the local RT is set as the baseline "Alert-RT". Driver risk response detection results for EEG signals with local and global RTs less than 1.5 times the Alert-RT are classified as high-risk responses, while those for EEG signals with local and global RTs greater than 2.5 times the Alert-RT are classified as low-risk responses.

[0094] In one optional embodiment of this application, the frequency-aware temporal convolutional model is trained using EEG samples carrying labels for high-risk or low-risk responses as training data. The training process specifically includes: first, inputting EEG samples carrying labels for high-risk or low-risk responses into the frequency-aware temporal convolutional model to be trained, and outputting the driver risk response detection results corresponding to the EEG samples; then, calculating the cross-entropy loss based on the driver risk response detection results and corresponding labels; and optimizing the frequency-aware temporal convolutional model to be trained using the calculated cross-entropy loss to obtain the aforementioned frequency-aware temporal convolutional model. The formula for calculating the cross-entropy loss is as follows:

[0095]

[0096] In the formula, y j This represents the true label of the EEG sample. This indicates the driver risk response test results corresponding to the EEG sample.

[0097] The first embodiment of this application proposes a driver risk response detection method based on a frequency-aware temporal convolution model. This method involves preprocessing the current EEG signal acquired by EEG signal electrodes to reduce noise, resulting in a preprocessed EEG signal. This preprocessed EEG signal is then input into a frequency-aware temporal convolution model. From a temporal perspective, the model's spatial and temporal feature extraction modules employ time series analysis to sequentially extract spatial and temporal features, yielding spatiotemporal features. The model's frequency-aware enhancement module converts the preprocessed EEG signal into a frequency domain sequence, and frequency analysis is used to enhance its features and extract enhanced features. Finally, the spatiotemporal features and the enhanced features are input into the model's fusion classification module to obtain the driver risk response detection result. This result characterizes the timeliness of a driver's response to potential dangers or emergencies during driving. This application proposes a frequency-aware temporal convolution model based on joint analysis of time series and frequency, and applies it to detect drivers' risk responsiveness, which improves detection accuracy, helps to study human factors in driving, and also helps to reduce traffic accidents.

[0098] In a second aspect of this application, a driver risk response detection device based on a frequency-aware temporal convolution model is provided, see [reference]. Figure 5 , Figure 5This is an architectural diagram of a driver risk response detection device based on a frequency-aware temporal convolution model provided in an embodiment of this application. The device includes:

[0099] The preprocessing module 501 is used to perform noise reduction preprocessing on the current EEG signal acquired by the EEG signal electrode to obtain the noise-reduced current EEG signal.

[0100] Input module 502 is used to input the noise-reduced preprocessed current EEG signal into the frequency-aware temporal convolution model;

[0101] The spatiotemporal feature extraction module 503 is used to extract spatial and temporal features from the current EEG signal after noise reduction and preprocessing in the time domain dimension through the spatial feature extraction module and temporal feature extraction module of the frequency-aware temporal convolution model, and to obtain spatiotemporal features by using time series analysis methods.

[0102] The enhanced feature extraction module 504 is used to convert the current EEG signal after noise reduction preprocessing into a frequency domain sequence through the frequency-aware enhancement module of the frequency-aware temporal convolution model, perform feature enhancement from the frequency domain dimension using frequency analysis methods, and extract enhanced features.

[0103] The detection result acquisition module 505 is used to input the spatiotemporal features and the enhanced features into the fusion classification module of the frequency-aware temporal convolution model to obtain the driver risk response detection result. The driver risk response detection result is used to characterize the timeliness of the driver's response to potential dangers or emergencies during driving.

[0104] The frequency-aware enhancement module in the enhanced feature extraction module includes a first transform layer, a low-pass filter layer, a fully connected linear layer, a zero-padding layer, an inverse transform layer, and a linear layer; the enhanced feature extraction module includes:

[0105] The first transformation submodule is used to process the input noise-reduced preprocessed current EEG signal through the first transformation layer to obtain the frequency domain sequence;

[0106] The processing submodule is used to pass the frequency domain sequence sequentially through the low-pass filter layer, the fully connected linear layer and the zero-padding layer to obtain the processed frequency domain sequence.

[0107] The feature acquisition submodule is used to obtain the enhanced features by sequentially passing the processed frequency domain sequence through the inverse transform layer and the linear layer.

[0108] Specifically, the enhanced feature extraction module includes:

[0109] The second transformation submodule is used to perform channel normalization processing on the input noise-reduced preprocessed current EEG signal, and to use the first transformation layer to transform the normalized current EEG signal from the time domain dimension to the frequency domain dimension to obtain the frequency domain sequence.

[0110] The truncation submodule is used to truncate the frequency domain sequence through the low-pass filter layer to obtain filtered frequency band data.

[0111] The sampling and zero-padding submodule is used to upsample the filtered frequency band data using the fully connected linear layer, and then pad the high-frequency part of the upsampled frequency band data with zeros using the zero-padding layer to obtain the processed frequency domain data.

[0112] The third transformation submodule is used to convert the processed frequency domain data from the frequency domain dimension into an EEG signal in the time domain dimension through the inverse transformation layer;

[0113] The dimensionality reduction submodule is used to perform dimensionality reduction processing on the time-domain EEG signal through the linear layer and extract the enhanced features.

[0114] The spatial feature extraction module in the spatiotemporal feature extraction module includes a first 2D convolutional layer and a first batch of normalization layers. The size of the convolution kernel in the first 2D convolutional layer is set to channel*1, where channel represents the channel of the convolution kernel in the first 2D convolutional layer, so as to perform spatial convolution on signals of different channels along the time series.

[0115] The spatiotemporal feature extraction module includes:

[0116] The spatial convolution submodule is used to input the current EEG signal after noise reduction preprocessing into the first 2D convolutional layer to perform spatial convolution and obtain the spatial convolution result.

[0117] The first normalization submodule is used to input the spatial convolution result into the first batch of normalization layers to obtain the spatial features.

[0118] The temporal feature extraction module in the spatiotemporal feature extraction module is connected after the spatial feature extraction module. The temporal feature extraction module includes: a second 2D convolutional layer, a second batch normalization layer, a nonlinear activation layer, and a max pooling layer. The size of the convolutional kernel in the second 2D convolutional layer is set to 1*length, where length represents the length of the convolutional kernel in the second 2D convolutional layer.

[0119] The spatiotemporal feature extraction module also includes:

[0120] The temporal convolution submodule is used to input the spatial features into the second 2D convolutional layer, perform temporal convolution, and obtain the temporal convolution result.

[0121] The second normalization submodule is used to input the temporal convolution result into the second batch of normalization layers to obtain the first spatiotemporal feature;

[0122] The nonlinearization processing submodule is used to input the first spatiotemporal feature into the nonlinear activation layer, perform nonlinearization processing, and obtain the second spatiotemporal feature;

[0123] The spatiotemporal feature acquisition submodule is used to input the second spatiotemporal feature into the max pooling layer for enhancement processing to obtain the spatiotemporal feature.

[0124] The preprocessing module includes:

[0125] The noise reduction submodule is used to perform noise reduction preprocessing on the current EEG signal acquired by the EEG signal electrodes through a 1-50Hz bandpass filter.

[0126] The frequency-aware temporal convolutional model in the input module is trained using EEG samples carrying labels for high-risk or low-risk responses as training data. The device also includes a training module, which comprises:

[0127] The detection result acquisition submodule is used to input EEG samples carrying high-risk response or low-risk response labels into the frequency-aware temporal convolutional model to be trained, and output the driver risk response detection result corresponding to the EEG sample.

[0128] The loss calculation submodule is used to calculate the cross-entropy loss based on the driver risk response detection results and corresponding labels of the EEG samples;

[0129] The optimization submodule is used to optimize the frequency-aware temporal convolutional model to be trained using the cross-entropy loss, so as to obtain the frequency-aware temporal convolutional model.

[0130] Based on the same concept, this application discloses an electronic device in a third aspect. Figure 6 A schematic diagram of an electronic device disclosed in an embodiment of this application is shown, such as... Figure 6 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory of the electronic device is not less than 12G, and the main frequency of the processor is not less than 2.4GHz. The memory 110 and the processor 120 are connected via a bus communication connection. The memory 110 stores a computer program, which can run on the processor 120 to implement a driver risk response detection method based on a frequency-aware temporal convolution model disclosed in the embodiments of this application.

[0131] Based on the same concept, this application discloses a computer-readable storage medium storing a computer program / instructions thereon in a fourth aspect. When the computer program / instructions are executed by a processor, they implement a driver risk response detection method based on a frequency-aware temporal convolution model disclosed in this application.

[0132] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0133] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0134] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0136] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0137] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0138] The foregoing has provided a detailed description of the driver risk response detection method and apparatus based on a frequency-aware temporal convolution model provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A driver risk response detection method based on a frequency-aware temporal convolution model, characterized in that, The method includes: The current EEG signal acquired by the EEG signal electrode is subjected to noise reduction preprocessing to obtain the current EEG signal after noise reduction preprocessing; The current EEG signal after noise reduction preprocessing is input into the frequency-aware temporal convolution model; From the time domain dimension, through the spatial feature extraction module and the temporal feature extraction module of the frequency-aware temporal convolution model, the time series analysis method is used to sequentially extract spatial and temporal features from the current EEG signal after noise reduction and preprocessing to obtain spatiotemporal features. The frequency-aware enhancement module of the frequency-aware temporal convolution model converts the current EEG signal after noise reduction preprocessing into a frequency domain sequence, performs feature enhancement from the frequency domain dimension using frequency analysis methods, and extracts the enhanced features. The spatiotemporal features and the enhanced features are input into the fusion classification module of the frequency-aware temporal convolution model to obtain the driver risk response detection result. The driver risk response detection result is used to characterize the timeliness of the driver's response to potential dangers or emergencies during driving. The frequency-aware enhancement module includes a first transform layer, a low-pass filter layer, a fully connected linear layer, a zero-padding layer, an inverse transform layer, and a linear layer. Through the frequency-aware enhancement module of the frequency-aware temporal convolution model, the current EEG signal after noise reduction preprocessing is converted into a frequency domain sequence. Feature enhancement is performed from the frequency domain dimension using frequency analysis methods, and enhanced features are extracted, including: The input noise-reduced preprocessed current EEG signal is passed through the first transform layer to obtain the frequency domain sequence; The frequency domain sequence is sequentially passed through the low-pass filter layer, the fully connected linear layer, and the zero-padding layer to obtain the processed frequency domain sequence. The processed frequency domain sequence is sequentially passed through the inverse transform layer and the linear layer to obtain the enhanced features; The frequency-aware enhancement module of the frequency-aware temporal convolution model converts the noise-reduced preprocessed EEG signal into a frequency domain sequence, performs feature enhancement using frequency analysis methods in the frequency domain, and extracts enhanced features, specifically including: The input EEG signal after noise reduction preprocessing is subjected to channel normalization processing, and the normalized current EEG signal is transformed from the time domain dimension to the frequency domain dimension using the first transform layer to obtain the frequency domain sequence. The frequency domain sequence is truncated by the low-pass filter layer to obtain filtered frequency band data. After upsampling the filtered frequency band data using the fully connected linear layer, the high-frequency portion of the upsampled frequency band data is padded with zeros using the zero-padding layer to obtain the processed frequency domain data. The inverse transform layer converts the processed frequency domain data from the frequency domain dimension into an EEG signal in the time domain dimension. The linear layer is used to reduce the dimensionality of the time-domain EEG signal and extract the enhanced features.

2. The driver risk response detection method based on a frequency-aware temporal convolution model according to claim 1, characterized in that, The spatial feature extraction module includes a first 2D convolutional layer and a first batch of normalization layers. The size of the convolution kernel in the first 2D convolutional layer is set to channel*1, where channel represents the channel of the convolution kernel in the first 2D convolutional layer, so as to perform spatial convolution on signals of different channels along the time series. Spatial feature extraction is performed on the current EEG signal after noise reduction preprocessing, including: The current EEG signal after noise reduction preprocessing is input into the first 2D convolutional layer to perform spatial convolution and obtain the spatial convolution result; The spatial convolution result is input into the first batch of normalization layers to obtain the spatial features.

3. The driver risk response detection method based on a frequency-aware temporal convolution model according to claim 2, characterized in that, The temporal feature extraction module is connected after the spatial feature extraction module. The temporal feature extraction module includes: a second 2D convolutional layer, a second batch normalization layer, a non-linear activation layer, and a max pooling layer. The size of the convolutional kernel in the second 2D convolutional layer is set to 1*length, where length represents the length of the convolutional kernel in the second 2D convolutional layer. Extracting time features from the current EEG signal after noise reduction preprocessing includes: The spatial features are input into the second 2D convolutional layer and temporal convolution is performed to obtain the temporal convolution result; The temporal convolution result is input into the second batch of normalization layers to obtain the first spatiotemporal feature; The first spatiotemporal feature is input into the nonlinear activation layer and subjected to nonlinearization processing to obtain the second spatiotemporal feature. The second spatiotemporal feature is input into the max pooling layer for enhancement processing to obtain the spatiotemporal feature.

4. The driver risk response detection method based on a frequency-aware temporal convolution model according to claim 1, characterized in that, The current EEG signal acquired by the EEG signal electrodes is preprocessed for noise reduction using a bandpass filter of 1-50 Hz.

5. The driver risk response detection method based on a frequency-aware temporal convolution model according to claim 1, characterized in that, The frequency-aware temporal convolutional model is trained using EEG samples carrying labels for high-risk or low-risk responses as training data. The training process of the frequency-aware temporal convolutional model includes: By inputting EEG samples carrying high-risk response or low-risk response labels into the frequency-aware temporal convolutional model to be trained, the driver risk response detection results corresponding to the EEG samples are output. Based on the driver risk response detection results and corresponding labels of the EEG samples, calculate the cross-entropy loss; The frequency-aware temporal convolutional model to be trained is optimized by the cross-entropy loss to obtain the frequency-aware temporal convolutional model.

6. A driver risk response detection device based on a frequency-aware temporal convolution model, characterized in that, The device includes: The preprocessing module is used to perform noise reduction preprocessing on the current EEG signal acquired by the EEG signal electrodes to obtain the noise-reduced current EEG signal; The input module is used to input the noise-reduced preprocessed current EEG signal into the frequency-aware temporal convolution model; The spatiotemporal feature extraction module is used to extract spatiotemporal features from the time domain dimension by using the spatial feature extraction module and the temporal feature extraction module of the frequency-aware temporal convolution model, and by employing time series analysis methods to sequentially extract spatial and temporal features from the current EEG signal after noise reduction and preprocessing, thereby obtaining spatiotemporal features. The enhanced feature extraction module is used to convert the current EEG signal after noise reduction preprocessing into a frequency domain sequence through the frequency-aware enhancement module of the frequency-aware temporal convolution model, perform feature enhancement from the frequency domain dimension using frequency analysis methods, and extract enhanced features. The detection result acquisition module is used to input the spatiotemporal features and the enhanced features into the fusion classification module of the frequency-aware temporal convolution model to obtain the driver risk response detection result. The driver risk response detection result is used to characterize the timeliness of the driver's response to potential dangers or emergencies during driving. The frequency sensing enhancement module includes a first transform layer, a low-pass filter layer, a fully connected linear layer, a zero-padding layer, an inverse transform layer, and a linear layer; the enhanced feature extraction module is specifically used for: The input noise-reduced preprocessed current EEG signal is passed through the first transform layer to obtain the frequency domain sequence; The frequency domain sequence is sequentially passed through the low-pass filter layer, the fully connected linear layer, and the zero-padding layer to obtain the processed frequency domain sequence. The processed frequency domain sequence is sequentially passed through the inverse transform layer and the linear layer to obtain the enhanced features; The enhanced feature extraction module is further used for: The input EEG signal after noise reduction preprocessing is subjected to channel normalization processing, and the normalized current EEG signal is transformed from the time domain dimension to the frequency domain dimension using the first transform layer to obtain the frequency domain sequence. The frequency domain sequence is truncated by the low-pass filter layer to obtain filtered frequency band data. After upsampling the filtered frequency band data using the fully connected linear layer, the high-frequency portion of the upsampled frequency band data is padded with zeros using the zero-padding layer to obtain the processed frequency domain data. The inverse transform layer converts the processed frequency domain data from the frequency domain dimension into an EEG signal in the time domain dimension. The linear layer is used to reduce the dimensionality of the time-domain EEG signal and extract the enhanced features.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the driver risk response detection method based on the frequency-aware temporal convolution model as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a computer program / instruction, which, when executed by a processor, implements the driver risk response detection method based on the frequency-aware temporal convolution model as described in any one of claims 1-5.

Images