Driver risk response detection method and device based on frequency perception time convolution model

The frequency-aware time convolution model addresses the limitations of existing methods by integrating time series and frequency analysis for EEG-based driver risk response detection, enhancing feature extraction and classification to improve detection accuracy and safety.

CN119066496AActive Publication Date: 2024-12-03TSINGHUA UNIVERSITY
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411209358.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-12-03
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

Existing driver risk response detection methods fail to comprehensively consider time series signals and frequency domain characteristics, leading to low detection accuracy in assessing a driver's ability to respond to potential dangers or emergencies.

Method used

A frequency-aware time convolution model is employed to process EEG signals, incorporating noise reduction preprocessing, spatial and temporal feature extraction, and frequency domain enhancement to improve detection accuracy.

Benefits of technology

The model enhances feature extraction and classification, improving the timeliness assessment of driver responses to emergencies, thereby reducing traffic accidents.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066496A_ABST
    Figure CN119066496A_ABST
Patent Text Reader

Abstract

The invention provides a driver risk response detection method and device based on a frequency sensing time convolution model, and relates to the technical field of data processing, and the method comprises the steps: carrying out the noise reduction preprocessing of a collected current EEG signal, and inputting the current EEG signal after noise reduction preprocessing into the frequency sensing time convolution model; sequentially performing spatial feature extraction and time feature extraction on the current EEG signal subjected to noise reduction preprocessing through a spatial feature extraction module and a time feature extraction module to obtain spatial and temporal features; the current EEG signal is converted into a frequency domain sequence through a frequency perception enhancement module, and enhancement features are extracted; and inputting the spatial-temporal features and the enhanced features into a fusion classification module to obtain a driver risk response detection result. The frequency perception time convolution model based on the time sequence and frequency conjoint analysis is provided and is applied to detecting the risk responsiveness of the driver, the detection precision is improved, human factors in driving can be researched, and traffic accidents can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a driver risk response detection method and device based on a frequency-aware time convolution model. Background Art

[0002] Driver risk response ability refers to the timeliness of the driver's response to potential dangers or emergencies, which has a significant impact on traffic safety. The driver's reaction time is crucial and is affected by various states such as distraction and fatigue. In the current mixed traffic context, whether humans directly handle emergencies as vehicle drivers or take over complex situations as safety supervisors of autonomous driving, a high degree of risk response ability is required to ensure traffic safety. Therefore, investigating the driver's risk response ability is a very important issue.

[0003] When a driver is drowsy or engaged in tasks unrelated to driving, their risk response ability is reduced, which may make it difficult for them to avoid emergency situations. Current research in this field focuses on monitoring human body data, which can assess the driver's current risk response ability, but image-based research is affected by external factors such as lighting and occlusion. Therefore, many studies use physiological signals for investigation, such as EEG signals (electroencephalogram signals).

[0004] However, existing EEG processing and modeling methods have certain limitations. Summary of the invention

[0005] The present application provides a driver risk response detection method and device based on a frequency-aware time convolution model to solve the problem that the existing driver risk response detection method cannot comprehensively consider time series signals and frequency domain characteristics, resulting in low detection accuracy.

[0006] In a first aspect of an embodiment of the present application, a driver risk response detection method based on a frequency-aware time convolution model is proposed, the method comprising:

[0007] Performing noise reduction preprocessing on the current EEG signal collected by the EEG signal electrode to obtain the current EEG signal after noise reduction preprocessing;

[0008] Inputting the current EEG signal after the noise reduction preprocessing into the frequency-aware time convolution model;

[0009] From the time domain dimension, the spatial feature extraction module and the temporal feature extraction module of the frequency-aware time convolution model are used to extract spatial features and temporal features of the current EEG signal after the noise reduction preprocessing in turn by using a time series analysis method to obtain spatiotemporal features;

[0010] The current EEG signal after the noise reduction preprocessing is converted into a frequency domain sequence through the frequency perception enhancement module of the frequency perception time convolution model, and the frequency analysis method is used to enhance the features from the frequency domain dimension, and the enhanced features are extracted;

[0011] The spatiotemporal features and the enhanced features are input into a fusion classification module of the frequency-aware temporal convolution model to obtain a driver risk response detection result, wherein the driver risk response detection result is used to characterize the timeliness of the driver's response to potential dangers or emergencies during driving.

[0012] In an optional embodiment of the present application, the frequency perception enhancement module includes a first transform layer, a low-pass filter layer, a fully connected linear layer, a zero-padding layer, an inverse transform layer and a linear layer;

[0013] The current EEG signal after the noise reduction preprocessing is converted into a frequency domain sequence through the frequency perception enhancement module of the frequency perception time convolution model, and the frequency analysis method is used to enhance the features from the frequency domain dimension, and the enhanced features are extracted, including:

[0014] The input current EEG signal after the noise reduction preprocessing is passed through the first transformation layer to obtain the frequency domain sequence;

[0015] The frequency domain sequence passes through the low-pass filtering layer, the fully connected linear layer and the zero-padding layer in sequence to obtain a processed frequency domain sequence;

[0016] The processed frequency domain sequence passes through the inverse transform layer and the linear layer in sequence to obtain the enhanced features.

[0017] In an optional embodiment of the present application, the current EEG signal after the noise reduction preprocessing is converted into a frequency domain sequence through the frequency perception enhancement module of the frequency perception time convolution model, and the frequency analysis method is used to enhance the features from the frequency domain dimension, and the enhanced features are extracted, which specifically includes:

[0018] Performing channel normalization processing on the input current EEG signal after the noise reduction preprocessing, and using the first transformation layer to transform the current EEG signal after the normalization processing from the time domain dimension to the frequency domain dimension to obtain the frequency domain sequence;

[0019] Performing frequency band interception on the frequency domain sequence through the low-pass filter layer to obtain filtered frequency band data;

[0020] After upsampling the filtered frequency band data by using the fully connected linear layer, the high frequency part of the upsampled frequency band data is padded with zeros by using the zero-padding layer to obtain processed frequency domain data;

[0021] The processed frequency domain data is converted from the frequency domain dimension to the EEG signal of the time domain dimension by the inverse transformation layer;

[0022] The EEG signal in the time domain dimension is subjected to dimensionality reduction processing through the linear layer, and the enhanced features are extracted.

[0023] In an optional embodiment of the present application, the spatial feature extraction module includes a first 2D convolution layer and a first batch of normalization layers, and the size of the convolution kernel in the first 2D convolution layer is set to: channel*1, where channel represents the channel of the convolution kernel in the first 2D convolution layer, so as to perform spatial convolution on different channel signals along the time series;

[0024] Performing spatial feature extraction on the current EEG signal after the noise reduction preprocessing, including:

[0025] Inputting the current EEG signal after the noise reduction preprocessing into the first 2D convolution layer to perform spatial convolution to obtain a spatial convolution result;

[0026] The spatial convolution results are input into the first batch of normalization layers to obtain the spatial features.

[0027] In an optional embodiment of the present application, the temporal feature extraction module is connected after the spatial feature extraction module, and the temporal feature extraction module includes: a second 2D convolution layer, a second batch normalization layer, a nonlinear activation layer and a maximum pooling layer, and the size of the convolution kernel in the second 2D convolution layer is set to: 1*length, where length represents the length of the convolution kernel in the second 2D convolution layer;

[0028] Extracting time features of the current EEG signal after the noise reduction preprocessing, including:

[0029] Inputting the spatial feature into the second 2D convolution layer, performing temporal convolution, and obtaining a temporal convolution result;

[0030] Inputting the temporal convolution result into the second batch normalization layer to obtain a first spatiotemporal feature;

[0031] Inputting the first spatiotemporal feature into the nonlinear activation layer for nonlinear processing to obtain a second spatiotemporal feature;

[0032] The second spatiotemporal feature is input into the maximum pooling layer and enhanced to obtain the spatiotemporal feature.

[0033] In an optional embodiment of the present application, the current EEG signal collected by the EEG signal electrode is pre-processed for noise reduction through a 1-50 Hz bandpass filter.

[0034] In an optional embodiment of the present application, the frequency-aware time convolution model is obtained by training with EEG samples carrying high-risk response or low-risk response labels as training data, and the training process of the frequency-aware time convolution model includes:

[0035] By inputting EEG samples carrying high-risk response or low-risk response labels into the frequency-aware temporal convolution model to be trained, outputting the driver risk response detection results corresponding to the EEG samples;

[0036] Calculating a cross entropy loss based on the driver risk response detection result corresponding to the EEG sample and the corresponding label;

[0037] The frequency-aware time convolution model to be trained is optimized by using the cross entropy loss to obtain the frequency-aware time convolution model.

[0038] In a second aspect of an embodiment of the present application, a driver risk response detection device based on a frequency-aware time convolution model is proposed, the device comprising:

[0039] A preprocessing module is used to perform noise reduction preprocessing on the current EEG signal collected by the EEG signal electrode to obtain the current EEG signal after noise reduction preprocessing;

[0040] An input module, used for inputting the current EEG signal after the noise reduction preprocessing into the frequency-aware time convolution model;

[0041] A spatiotemporal feature extraction module is used to extract spatial features and temporal features of the current EEG signal after the noise reduction preprocessing in sequence from the time domain dimension through the spatial feature extraction module and the temporal feature extraction module of the frequency-aware time convolution model, and adopt a time series analysis method to obtain spatiotemporal features;

[0042] An enhanced feature extraction module, used to convert the current EEG signal after the noise reduction preprocessing into a frequency domain sequence through the frequency perception enhancement module of the frequency perception time convolution model, perform feature enhancement from the frequency domain dimension using a frequency analysis method, and extract enhanced features;

[0043] The detection result acquisition module is used to input the spatiotemporal features and the enhanced features into the fusion classification module of the frequency-aware time convolution model to obtain the driver risk response detection result, and the driver risk response detection result is used to characterize the timeliness of the driver's response to potential dangers or emergencies during driving.

[0044] In a third aspect of an embodiment of the present application, an electronic device is proposed, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the driver risk response detection method based on the frequency-aware time convolution model as described in any one of the first aspects above.

[0045] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is proposed, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the driver risk response detection method based on the frequency-aware time convolution model described in any one of the first aspects above is implemented.

[0046] The present application includes the following advantages: a driver risk response detection method and device based on a frequency-aware time convolution model is proposed in the present application, by performing noise reduction preprocessing on the current EEG signal collected by the EEG signal electrode to obtain the current EEG signal after noise reduction preprocessing; the current EEG signal after noise reduction preprocessing is input into the frequency-aware time convolution model; from the time domain dimension, the spatial feature extraction module and the time feature extraction module of the frequency-aware time convolution model are used to extract spatial features and time features of the current EEG signal after noise reduction preprocessing in sequence by using a time series analysis method to obtain spatiotemporal features; the current EEG signal after noise reduction preprocessing is converted into a frequency domain sequence by using a frequency-aware enhancement module of the frequency-aware time convolution model, and feature enhancement is performed by using a frequency analysis method from the frequency domain dimension, and enhanced features are extracted; the spatiotemporal features and the enhanced features are input into the fusion classification module of the frequency-aware time convolution model to obtain a driver risk response detection result, and the driver risk response detection result is used to characterize the degree of timeliness of the driver's response to potential dangers or emergencies during driving. This application proposes a frequency-aware time convolution model based on joint analysis of time series and frequency, and applies it to detecting the risk responsiveness of drivers, thereby improving detection accuracy, helping to study human factors in driving, and also helping to reduce traffic accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 It is a schematic diagram of the steps of a driver risk response detection method based on a frequency-aware time convolution model proposed in an embodiment of the present application;

[0049] Figure 2It is a flowchart of a driver risk response detection method based on a frequency-aware time convolution model proposed in an embodiment of the present application;

[0050] Figure 3 Schematic diagram of the network structure of a frequency-aware time convolution model proposed in an embodiment of the present application;

[0051] Figure 4 It is a schematic diagram of a process of extracting enhanced features using a frequency-aware enhancement module proposed in an embodiment of the present application;

[0052] Figure 5 It is an architecture diagram of a driver risk response detection device based on a frequency-aware time convolution model provided in an embodiment of the present application;

[0053] Figure 6 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] Driver risk response ability refers to the timeliness of the driver's response to potential dangers or emergencies, which has a significant impact on traffic safety. The driver's reaction time is crucial and is affected by various states such as distraction and fatigue. In the current mixed traffic context, whether humans directly handle emergencies as vehicle drivers or take over complex situations as safety supervisors of autonomous driving, a high degree of risk response ability is required to ensure traffic safety. Therefore, investigating the driver's risk response ability is a very important issue.

[0056] When a driver is drowsy or engaged in non-driving related tasks (NDRTs), their risk response ability is reduced, which may make it difficult for them to avoid emergency situations. Current research in this field mainly focuses on the monitoring of human body data. For example, some studies detect the driver's drowsiness by analyzing the degree of eye closure and mouth opening in facial images. In addition, there are also studies that use bust images to detect whether a person is engaged in NDRTs, such as using a mobile phone or adjusting the radio. The above studies are able to assess the driver's current risk response ability. However, image-based studies are affected by external factors such as lighting and occlusion. Therefore, many studies use physiological signals for investigation, such as EEG signals (electroencephalogram signals).

[0057] However, existing EEG processing and modeling methods have certain limitations.

[0058] Based on this, the present application considers combining frequency perception with spatiotemporal convolution to construct a frequency-aware time convolution model that integrates the features of frequency and time series analysis. The model can convert time series data into the frequency domain, and use frequency analysis methods in the frequency domain for data processing and feature extraction. In the time domain, time series analysis methods can be used for modeling and prediction, and the model can be applied to EEG-based driver risk responsiveness detection.

[0059] In a first aspect of an embodiment of the present application, a driver risk response detection method based on a frequency-aware time convolution model is provided. Figure 1 , Figure 1 1 is a schematic diagram of a process flow of a driver risk response detection method based on a frequency-aware time convolution model proposed in an embodiment of the present application, the method comprising the following steps:

[0060] Step 101: performing noise reduction preprocessing on the current EEG signal collected by the EEG signal electrode to obtain the current EEG signal after noise reduction preprocessing;

[0061] Step 102: inputting the current EEG signal after the noise reduction preprocessing into a frequency-aware time convolution model;

[0062] Step 103: from the time domain dimension, the spatial feature extraction module and the temporal feature extraction module of the frequency-aware time convolution model are used to extract spatial features and temporal features of the current EEG signal after the noise reduction preprocessing in sequence by using a time series analysis method to obtain spatiotemporal features;

[0063] Step 104: converting the current EEG signal after the noise reduction preprocessing into a frequency domain sequence through the frequency perception enhancement module of the frequency perception time convolution model, performing feature enhancement from the frequency domain dimension using a frequency analysis method, and extracting enhanced features;

[0064] Step 105: Input the spatiotemporal features and the enhanced features into the fusion classification module of the frequency-aware temporal convolution model to obtain a driver risk response detection result, where the driver risk response detection result is used to characterize the timeliness of the driver's response to potential dangers or emergencies during driving.

[0065] In order to clearly explain the driver risk response detection method based on the frequency-aware time convolution model proposed in this application, Figure 2 For example, Figure 2It is a flow chart of a driver risk response detection method based on a frequency-aware time convolution model proposed in an embodiment of the present application.

[0066] When implementing step 101 specifically, the EEG signal collection method used in this application is: using an event-related lane departure paradigm, the participants are between 22 and 28 years old, and the participants are assigned continuous driving tasks in a virtual reality simulator, and random lane departure events are introduced. Once deviating from the central lane, the participants are required to drive the car back quickly and record the EEG signal reflecting the reaction time of the subjects. Among them, the EEG signal is collected using 30 EEG signal electrodes at a sampling rate of 500Hz. This application performs noise reduction preprocessing on the current EEG signal collected as above to obtain the current EEG signal after noise reduction preprocessing to eliminate the influence of noise on the accuracy of subsequent detection results.

[0067] In an optional embodiment of the present application, the collected current EEG signal is pre-processed for noise reduction by using a 1-50 Hz bandpass filter. The 1-50 Hz bandpass filter can remove high-frequency signals above 50 Hz of the current EEG signal, remove signal noise, and achieve noise reduction processing of the current EEG signal.

[0068] When the step 102 is specifically implemented, the current EEG signal (corresponding to Figure 2 After the noise reduction processing is performed on the Raw Signal in the image, the current EEG signal after noise reduction preprocessing (corresponding to Figure 2 Preprocessed Signal in) Input frequency-aware temporal convolution model (corresponding to Figure 2 The frequency and time characteristics of the current EEG signal are collected, and the timeliness of the driver's risk response is analyzed based on the collected frequency and time characteristics. The network structure of the above frequency-aware time convolution model can be found in 3, Figure 3 is a schematic diagram of a network structure of a frequency-aware time convolution model proposed in an embodiment of the present application. The frequency-aware time convolution model includes: a spatial feature extraction module (corresponding to Figure 3 Spatial feature extraction module used in ), temporal feature extraction module (corresponding to Figure 3 Temporal feature extraction module used in), frequency perception enhancement module (corresponding to Figure 3 The module used in Frequency aware enhancement) and the fusion classification module (corresponding to Figure 3 The module used in Fusion and classification).

[0069] When the step 103 is specifically implemented, the current EEG signal collected is a time series signal. Based on this, from the time domain dimension, the spatial feature extraction module and the time feature extraction module of the frequency-aware time convolution model are used to extract spatial features and time features of the current EEG signal after noise reduction preprocessing in turn using a time series analysis method to obtain spatiotemporal features. The time series analysis method refers to extracting spatial features and time features of the EEG signal using the spatial feature extraction module and the time feature extraction module of the frequency-aware time convolution model.

[0070] In an optional embodiment of the present application, since the EEG signal has spatial coupling, it can be regarded as a mixture of signals from different brain regions. For the current EEG signal input, the spatial feature extraction module is first used to extract its spatial features. The above-mentioned spatial feature extraction module includes a first 2D convolution layer and a first batch of normalization layers. The size of the convolution kernel in the above-mentioned first 2D convolution layer is set to: channel*1, channel represents the channel of the convolution kernel in the first 2D convolution layer, so as to perform spatial convolution on different channel signals along the time series. Specifically, the spatial feature extraction of the current EEG signal after the above-mentioned noise reduction preprocessing includes the following process: first, the current EEG signal after the above-mentioned noise reduction preprocessing is input into the above-mentioned first 2D convolution layer, and spatial convolution is performed to obtain a spatial convolution result; after the spatial convolution, the above-mentioned spatial convolution result is input into the above-mentioned first batch of normalization layers for normalization processing to promote the stability and convergence speed of the frequency-aware time convolution model and obtain spatial features. Among them, the calculation formula for normalization processing using the first batch of normalization layers is as follows:

[0071]

[0072] Where BN represents the first batch of normalization layers, x represents the spatial convolution result, μ and σ represent the mean and standard deviation of the current EEG signal after input denoising preprocessing, γ and β represent the learnable scaling parameter and offset parameter, respectively.

[0073] In an optional embodiment of the present application, after completing the spatial feature extraction, convolution is applied to extract the temporal features. The extraction of the temporal features is implemented by a temporal feature extraction module. The temporal feature extraction module is connected after the spatial feature extraction module. The temporal feature extraction module includes: a second 2D convolution layer, a second batch normalization layer, a nonlinear activation layer and a maximum pooling layer. The size of the convolution kernel in the second 2D convolution layer is set to: 1*length, where length represents the length of the convolution kernel in the second 2D convolution layer; temporal convolution is performed by configuring a convolution kernel with a specific length and an appropriate stride to facilitate the capture of temporal patterns in the data. Specifically, temporal feature extraction is performed on the current EEG signal after the above-mentioned noise reduction preprocessing, including the following process: inputting the above-mentioned spatial features into the above-mentioned second 2D convolution layer, performing temporal convolution, and obtaining a temporal convolution result; then inputting the above-mentioned temporal convolution result into the above-mentioned second batch normalization layer, performing normalization processing, and obtaining a first spatiotemporal feature; after obtaining the first spatiotemporal feature, the first spatiotemporal feature is input into the above-mentioned nonlinear activation layer (corresponding to Figure 3 The Relu in the above example is used to perform nonlinear processing to obtain the second spatiotemporal feature to enhance the representation ability of the frequency-aware temporal convolution model. Finally, the second spatiotemporal feature is input into the above maximum pooling layer for enhancement processing to obtain the spatiotemporal feature (corresponding to Figure 3 Spatial-temporal feature in .

[0074] When the step 104 is specifically implemented, after obtaining the spatiotemporal features, in order to enhance the features through frequency perception technology, the present application converts the current EEG signal after the noise reduction preprocessing into a frequency domain sequence through the frequency perception enhancement module of the frequency perception time convolution model, and enhances the features from the frequency domain dimension using the frequency analysis method, and extracts the enhanced features (corresponding to Figure 3 The above frequency analysis methods refer to Fourier transform (FFT), discrete Fourier transform (DFT), real fast Fourier transform (rFFT) and other methods.

[0075] In an optional embodiment of the present application, see Figure 4 , Figure 4It is a flow chart of a method of extracting enhanced features using a frequency-aware enhancement module proposed in an embodiment of the present application. The frequency-aware enhancement module includes a first transformation layer, a low-pass filter layer, a fully connected linear layer, a zero-padding layer, an inverse transformation layer, and a linear layer; the current EEG signal after noise reduction preprocessing is converted into a frequency domain sequence through the frequency-aware time convolution model's frequency-aware enhancement module, and feature enhancement is performed from the frequency domain dimension using a frequency analysis method, and enhanced features are extracted, including the following process: first, the input time series signal, that is, the current EEG signal after noise reduction preprocessing, is passed through the first transformation layer to obtain a frequency domain sequence; then, the frequency domain sequence is sequentially passed through the low-pass filter layer, the fully connected linear layer, and the zero-padding layer to obtain a processed frequency domain sequence; finally, the processed frequency domain sequence is sequentially passed through the inverse transformation layer and the linear layer to obtain enhanced features.

[0076] In another optional embodiment of the present application, the current EEG signal after noise reduction preprocessing is converted into a frequency domain sequence through a frequency perception enhancement module of a frequency perception time convolution model, and a frequency analysis method is used to enhance features from the frequency domain dimension, and enhanced features are extracted, specifically including the following processes:

[0077] First, the input time series signal (corresponding to Figure 4 The current EEG signal after noise reduction preprocessing is subjected to channel normalization processing, and the first transformation layer is used to transform the normalized current EEG signal from the time domain dimension to the frequency domain dimension to obtain the frequency domain sequence (corresponding to Figure 4 The conversion here is achieved through real fast Fourier transform (rFFT), which uses the symmetry of real-valued sequences in the frequency domain to convert complex-valued sequences in the frequency domain into half-length real-valued sequences. Specifically, for a real-valued sequence of length N, rFFT transforms it into a complex-valued sequence of length N / 2+1, encapsulating the spectral information of the sequence.

[0078] After the frequency domain transformation is performed by the first transformation layer, the obtained frequency sequence represents the components of the signal at different frequencies. Then, the low-pass filter layer is used to intercept the frequency band in the frequency domain sequence (corresponding to Figure 4 The LPF in the filter can suppress the side lobe interference after filtering and eliminate the interference of high-frequency signals. Low-pass filtering can be achieved by truncating the frequency domain sequence at a specific position. This truncation operation can be understood as eliminating the components above the cutoff position in the frequency domain to obtain the filtered frequency band data (corresponding to Figure 4 Low-frequncy in ).

[0079] For example, assuming the desired cutoff frequency is fcutoff , then the corresponding cutoff position k cutoff It can be calculated using the following formula:

[0080] In the formula, f s Represents the sampling rate of the frequency domain sequence.

[0081] The low-pass filter layer effectively removes the frequencies above the cut-off frequency f cutoff frequency components, thereby achieving the desired low-pass filtering effect.

[0082] Subsequently, a fully connected linear layer is used to upsample the filtered frequency band data (corresponding to Figure 4 LinearUp-sampling in ), the up-sampled frequency band data corresponds to Figure 4 In the Interpolated Sequence, upsampling aims to increase the sampling rate of the signal to make it more refined in the time domain. Linear interpolation is usually used to interpolate frequency domain sequences. In the frequency domain dimension, linear interpolation usually requires interpolation using adjacent points on the frequency axis.

[0083] For example, assuming that there are amplitude values ​​A(f1) and A(f2) corresponding to frequencies f1 and f2, and it is desired to interpolate at a frequency f between f1 and f2, the linear interpolation formula can be expressed as:

[0084]

[0085] Where A(f) represents the interpolation result at frequency f.

[0086] After upsampling, the high frequency part of the upsampled frequency band data is padded with zeros through the zero-padding layer (corresponding to Figure 4 Zero padding in to ensure the consistency and integrity of the output data, and obtain the processed frequency domain data (corresponding to Figure 4 in the Padding Sequence);

[0087] For example, k>k in the frequency domain sequence cutoff The sampling points are set to zero, that is:

[0088]

[0089] When k≤k cutoff The sampling points are not reset to zero.

[0090] Subsequently, the processed frequency domain data is converted from the frequency domain dimension to the EEG signal in the time domain dimension (corresponding to Figure 4Enhanced Sequence in the frequency domain), that is, converting the frequency domain data into a time series signal (EEG signal in the time domain dimension). Here, the inverse transform layer is implemented by irFFT (rFFT inverse transform).

[0091] Finally, the EEG signal in the time domain dimension is reduced in dimension by the linear layer (corresponding to Figure 4 Dimensionality Reduction in the image, and extract enhanced features, which are used for subsequent processing and analysis.

[0092] When step 105 is specifically implemented, after obtaining the enhanced features, the spatiotemporal features and the enhanced features are input into the fusion classification module of the frequency-aware time convolution model, the spatiotemporal features and the enhanced features are fused, and the fused features are classified to obtain the driver risk response detection result. This process is implemented through the fully connected layer, corresponding to Figure 2 The Contact layer, Linear layer, ReLU layer and Linear layer parts in .

[0093] In an optional embodiment of the present application, the above driver risk response detection result is used to characterize the degree of promptness of the driver's response to potential dangers or emergencies during driving. The driver risk response detection result includes a high risk response (corresponding to Figure 2 High Risk Responsiveness) and Low Risk Responsiveness (corresponding to Figure 2 LowRisk Responsiveness in the , high-risk response and low-risk response are determined in the following way: the driver's risk responsiveness is evaluated based on the driver's reaction time. The local reaction time (RT) of each input EEG signal refers to the time it takes the driver to react to the lane departure event. The global reaction time (global RT) represents the average reaction time within the 90-second window before the lane departure event occurs, and the 5th percentile of the local RT is set as the baseline "alert-RT". The driver risk response detection results corresponding to the EEG signals with local RT and global RT less than 1.5 times the alert-RT are classified as high-risk responses, and the driver risk response detection results corresponding to the EEG signals with local RT and global RT greater than 2.5 times the alert-RT are classified as low-risk responses.

[0094] In an optional embodiment of the present application, the frequency-aware time convolution model is obtained by training with EEG samples carrying high-risk response or low-risk response labels as training data, and the training process of the frequency-aware time convolution model specifically includes: first, by inputting EEG samples carrying high-risk response or low-risk response labels into the frequency-aware time convolution model to be trained, and outputting the driver risk response detection results corresponding to the EEG samples; then, based on the driver risk response detection results corresponding to the EEG samples and the corresponding labels, the cross-entropy loss is calculated; the frequency-aware time convolution model to be trained is optimized by the calculated cross-entropy loss to obtain the frequency-aware time convolution model. The calculation formula of the cross-entropy loss is as follows:

[0095]

[0096] In the formula, y j represents the true label of the EEG sample, Represents the driver risk response detection result corresponding to the EEG sample.

[0097] In the first embodiment of the present application, a driver risk response detection method based on a frequency-aware time convolution model is proposed, which performs noise reduction preprocessing on the current EEG signal collected by the EEG signal electrode to obtain the current EEG signal after noise reduction preprocessing; the current EEG signal after noise reduction preprocessing is input into the frequency-aware time convolution model; from the time domain dimension, the spatial feature extraction module and the time feature extraction module of the frequency-aware time convolution model are used to extract spatial features and time features of the current EEG signal after noise reduction preprocessing in turn by using a time series analysis method to obtain spatiotemporal features; the current EEG signal after noise reduction preprocessing is converted into a frequency domain sequence by using the frequency-aware enhancement module of the frequency-aware time convolution model, and feature enhancement is performed by using a frequency analysis method from the frequency domain dimension, and enhanced features are extracted; the spatiotemporal features and the enhanced features are input into the fusion classification module of the frequency-aware time convolution model to obtain a driver risk response detection result, which is used to characterize the degree of timeliness of the driver's response to potential dangers or emergencies during driving. This application proposes a frequency-aware time convolution model based on joint analysis of time series and frequency, and applies it to detecting the risk responsiveness of drivers, thereby improving detection accuracy, helping to study human factors in driving, and also helping to reduce traffic accidents.

[0098] In a second aspect of the embodiment of the present application, a driver risk response detection device based on a frequency-aware time convolution model is provided. Figure 5 , Figure 5: is an architecture diagram of a driver risk response detection device based on a frequency-aware time convolution model provided in an embodiment of the present application, the device comprising:

[0099] The preprocessing module 501 is used to perform noise reduction preprocessing on the current EEG signal collected by the EEG signal electrode to obtain the current EEG signal after noise reduction preprocessing;

[0100] An input module 502, used to input the current EEG signal after the noise reduction preprocessing into a frequency-aware time convolution model;

[0101] The spatiotemporal feature extraction module 503 is used to extract spatial features and temporal features of the current EEG signal after the noise reduction preprocessing in sequence from the time domain dimension through the spatial feature extraction module and the temporal feature extraction module of the frequency-aware time convolution model, and adopt a time series analysis method to obtain spatiotemporal features;

[0102] An enhanced feature extraction module 504 is used to convert the current EEG signal after the noise reduction preprocessing into a frequency domain sequence through the frequency perception enhancement module of the frequency perception time convolution model, perform feature enhancement from the frequency domain dimension using a frequency analysis method, and extract enhanced features;

[0103] The detection result acquisition module 505 is used to input the spatiotemporal features and the enhanced features into the fusion classification module of the frequency-aware time convolution model to obtain the driver risk response detection result, and the driver risk response detection result is used to characterize the timeliness of the driver's response to potential dangers or emergencies during driving.

[0104] The frequency perception enhancement module in the enhanced feature extraction module includes a first transformation layer, a low-pass filter layer, a fully connected linear layer, a zero-filling layer, an inverse transformation layer and a linear layer; the enhanced feature extraction module includes:

[0105] A first transformation submodule, configured to obtain the frequency domain sequence by passing the input current EEG signal after the noise reduction preprocessing through the first transformation layer;

[0106] A processing submodule, configured to sequentially pass the frequency domain sequence through the low-pass filtering layer, the fully connected linear layer and the zero-padding layer to obtain a processed frequency domain sequence;

[0107] The feature acquisition submodule is used to sequentially pass the processed frequency domain sequence through the inverse transform layer and the linear layer to obtain the enhanced feature.

[0108] Wherein, the enhanced feature extraction module specifically includes:

[0109] A second transformation submodule is used to perform channel normalization processing on the input current EEG signal after the noise reduction preprocessing, and use the first transformation layer to transform the normalized current EEG signal from the time domain dimension to the frequency domain dimension to obtain the frequency domain sequence;

[0110] A submodule for performing frequency band interception on the frequency domain sequence through the low-pass filter layer to obtain filtered frequency band data;

[0111] A sampling zero-padding submodule, configured to perform zero padding on a high frequency portion of the upsampled frequency band data by using the fully connected linear layer after upsampling the filtered frequency band data, to obtain processed frequency domain data;

[0112] A third transformation submodule, configured to transform the processed frequency domain data from the frequency domain dimension into an EEG signal of the time domain dimension through the inverse transformation layer;

[0113] The dimensionality reduction processing submodule is used to perform dimensionality reduction processing on the EEG signal in the time domain dimension through the linear layer and extract the enhanced features.

[0114] The spatial feature extraction module in the spatiotemporal feature extraction module includes a first 2D convolution layer and a first batch of normalization layers, and the size of the convolution kernel in the first 2D convolution layer is set to: channel*1, where channel represents the channel of the convolution kernel in the first 2D convolution layer, so as to perform spatial convolution on different channel signals along the time series;

[0115] The spatiotemporal feature extraction module comprises:

[0116] A spatial convolution submodule, used for inputting the current EEG signal after the noise reduction preprocessing into the first 2D convolution layer, performing spatial convolution, and obtaining a spatial convolution result;

[0117] The first normalization submodule is used to input the spatial convolution results into the first batch of normalization layers to obtain the spatial features.

[0118] The temporal feature extraction module in the spatiotemporal feature extraction module is connected after the spatial feature extraction module, and the temporal feature extraction module includes: a second 2D convolution layer, a second batch normalization layer, a nonlinear activation layer and a maximum pooling layer, and the size of the convolution kernel in the second 2D convolution layer is set to: 1*length, where length represents the length of the convolution kernel in the second 2D convolution layer;

[0119] The spatiotemporal feature extraction module also includes:

[0120] A temporal convolution submodule, used for inputting the spatial features into the second 2D convolution layer, performing temporal convolution, and obtaining a temporal convolution result;

[0121] A second normalization submodule, used for inputting the temporal convolution result into the second batch normalization layer to obtain a first spatiotemporal feature;

[0122] A nonlinear processing submodule, used for inputting the first spatiotemporal feature into the nonlinear activation layer, performing nonlinear processing, and obtaining a second spatiotemporal feature;

[0123] The spatiotemporal feature acquisition submodule is used to input the second spatiotemporal feature into the maximum pooling layer, perform enhancement processing, and obtain the spatiotemporal feature.

[0124] Wherein, the preprocessing module includes:

[0125] The denoising submodule is used to perform denoising preprocessing on the current EEG signal collected by the EEG signal electrode through a 1-50 Hz bandpass filter.

[0126] The frequency-aware temporal convolution model in the input module is obtained by training with EEG samples carrying high-risk response or low-risk response labels as training data, and the device also includes a training module, which includes:

[0127] A detection result acquisition submodule, configured to input an EEG sample carrying a high-risk response or a low-risk response label into a frequency-aware time convolution model to be trained, and output a driver risk response detection result corresponding to the EEG sample;

[0128] A loss calculation submodule, used to calculate the cross entropy loss based on the driver risk response detection result corresponding to the EEG sample and the corresponding label;

[0129] An optimization submodule is used to optimize the frequency-aware time convolution model to be trained through the cross entropy loss to obtain the frequency-aware time convolution model.

[0130] Based on the same application concept, the embodiment of the present application discloses an electronic device in a third aspect. Figure 6 A schematic diagram of an electronic device disclosed in an embodiment of the present application is shown. Figure 6 As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory of the electronic device is not less than 12G, the main frequency of the processor is not less than 2.4GHz, the memory 110 and the processor 120 are connected via a bus communication, the memory 110 stores a computer program, and the computer program can be run on the processor 120 to implement a driver risk response detection method based on a frequency-aware time convolution model disclosed in an embodiment of the present application.

[0131] Based on the same application concept, an embodiment of the present application discloses in a fourth aspect a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, a driver risk response detection method based on a frequency-aware time convolution model disclosed in an embodiment of the present application is implemented.

[0132] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0133] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0134] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0136] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0137] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0138] The above is a detailed introduction to the driver risk response detection method and device based on the frequency-aware time convolution model provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A driver risk response detection method based on a frequency-aware temporal convolution model, characterized in that: The method comprises: Performing noise reduction preprocessing on the current EEG signal collected by the EEG signal electrode to obtain the current EEG signal after noise reduction preprocessing; Inputting the current EEG signal after the noise reduction preprocessing into the frequency-aware time convolution model; From the time domain dimension, the spatial feature extraction module and the temporal feature extraction module of the frequency-aware time convolution model are used to extract spatial features and temporal features of the current EEG signal after the noise reduction preprocessing in turn by using a time series analysis method to obtain spatiotemporal features; The current EEG signal after the noise reduction preprocessing is converted into a frequency domain sequence through the frequency perception enhancement module of the frequency perception time convolution model, and the frequency analysis method is used to enhance the features from the frequency domain dimension, and the enhanced features are extracted; The spatiotemporal features and the enhanced features are input into a fusion classification module of the frequency-aware temporal convolution model to obtain a driver risk response detection result, wherein the driver risk response detection result is used to characterize the timeliness of the driver's response to potential dangers or emergencies during driving.

2. The driver risk response detection method based on frequency-aware time convolution model according to claim 1 is characterized in that: The frequency perception enhancement module includes a first transform layer, a low-pass filter layer, a fully connected linear layer, a zero-padding layer, an inverse transform layer and a linear layer; The current EEG signal after the noise reduction preprocessing is converted into a frequency domain sequence through the frequency perception enhancement module of the frequency perception time convolution model, and the frequency analysis method is used to enhance the features from the frequency domain dimension, and the enhanced features are extracted, including: The input current EEG signal after the noise reduction preprocessing is passed through the first transformation layer to obtain the frequency domain sequence; The frequency domain sequence passes through the low-pass filtering layer, the fully connected linear layer and the zero-padding layer in sequence to obtain a processed frequency domain sequence; The processed frequency domain sequence passes through the inverse transform layer and the linear layer in sequence to obtain the enhanced features.

3. The driver risk response detection method based on frequency-aware time convolution model according to claim 2 is characterized in that: The frequency-aware enhancement module of the frequency-aware time convolution model is used to convert the current EEG signal after the noise reduction preprocessing into a frequency domain sequence, and the frequency analysis method is used to enhance the features from the frequency domain dimension, and the enhanced features are extracted, which specifically includes: Performing channel normalization processing on the input current EEG signal after the noise reduction preprocessing, and using the first transformation layer to transform the current EEG signal after the normalization processing from the time domain dimension to the frequency domain dimension to obtain the frequency domain sequence; Performing frequency band interception on the frequency domain sequence through the low-pass filter layer to obtain filtered frequency band data; After upsampling the filtered frequency band data by using the fully connected linear layer, the high frequency part of the upsampled frequency band data is padded with zeros by using the zero-padding layer to obtain processed frequency domain data; The processed frequency domain data is converted from the frequency domain dimension to the EEG signal of the time domain dimension by the inverse transformation layer; The EEG signal in the time domain dimension is subjected to dimensionality reduction processing through the linear layer, and the enhanced features are extracted.

4. The driver risk response detection method based on frequency-aware time convolution model according to claim 1 is characterized in that: The spatial feature extraction module includes a first 2D convolution layer and a first batch of normalization layers, the size of the convolution kernel in the first 2D convolution layer is set to: channel*1, channel represents the channel of the convolution kernel in the first 2D convolution layer, so as to perform spatial convolution on different channel signals along the time series; Performing spatial feature extraction on the current EEG signal after the noise reduction preprocessing, including: Inputting the current EEG signal after the noise reduction preprocessing into the first 2D convolution layer to perform spatial convolution to obtain a spatial convolution result; The spatial convolution results are input into the first batch of normalization layers to obtain the spatial features.

5. The driver risk response detection method based on frequency-aware time convolution model according to claim 4 is characterized in that: The temporal feature extraction module is connected after the spatial feature extraction module, and the temporal feature extraction module includes: a second 2D convolution layer, a second batch normalization layer, a nonlinear activation layer and a maximum pooling layer, and the size of the convolution kernel in the second 2D convolution layer is set to: 1*length, where length represents the length of the convolution kernel in the second 2D convolution layer; Extracting time features of the current EEG signal after the noise reduction preprocessing, including: Inputting the spatial feature into the second 2D convolution layer, performing temporal convolution, and obtaining a temporal convolution result; Inputting the temporal convolution result into the second batch normalization layer to obtain a first spatiotemporal feature; Inputting the first spatiotemporal feature into the nonlinear activation layer for nonlinear processing to obtain a second spatiotemporal feature; The second spatiotemporal feature is input into the maximum pooling layer and enhanced to obtain the spatiotemporal feature.

6. The driver risk response detection method based on frequency-aware time convolution model according to claim 1 is characterized in that: The current EEG signal collected by the EEG signal electrode is pre-processed for noise reduction through a 1-50 Hz bandpass filter.

7. The driver risk response detection method based on frequency-aware time convolution model according to claim 1 is characterized in that: The frequency-aware temporal convolutional model is obtained by training with EEG samples carrying high-risk response or low-risk response labels as training data. The training process of the frequency-aware temporal convolutional model includes: By inputting EEG samples carrying high-risk response or low-risk response labels into the frequency-aware temporal convolution model to be trained, outputting the driver risk response detection results corresponding to the EEG samples; Calculating a cross entropy loss based on the driver risk response detection result corresponding to the EEG sample and the corresponding label; The frequency-aware time convolution model to be trained is optimized by using the cross entropy loss to obtain the frequency-aware time convolution model.

8. A driver risk response detection device based on a frequency-aware time convolution model, characterized in that: The device comprises: A preprocessing module is used to perform noise reduction preprocessing on the current EEG signal collected by the EEG signal electrode to obtain the current EEG signal after noise reduction preprocessing; An input module, used for inputting the current EEG signal after the noise reduction preprocessing into the frequency-aware time convolution model; A spatiotemporal feature extraction module is used to extract spatial features and temporal features of the current EEG signal after the noise reduction preprocessing in sequence from the time domain dimension through the spatial feature extraction module and the temporal feature extraction module of the frequency-aware time convolution model, and adopt a time series analysis method to obtain spatiotemporal features; An enhanced feature extraction module, used to convert the current EEG signal after the noise reduction preprocessing into a frequency domain sequence through the frequency perception enhancement module of the frequency perception time convolution model, perform feature enhancement from the frequency domain dimension using a frequency analysis method, and extract enhanced features; The detection result acquisition module is used to input the spatiotemporal features and the enhanced features into the fusion classification module of the frequency-aware time convolution model to obtain the driver risk response detection result, and the driver risk response detection result is used to characterize the timeliness of the driver's response to potential dangers or emergencies during driving.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the driver risk response detection method based on the frequency-aware time convolution model described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the driver risk response detection method based on a frequency-aware time convolution model described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Depression electroencephalogram classification system based on multi-band space-time convolution network

    CN113723557A

  • Vehicle decision-making method and device based on electroencephalogram signals of passengers and computer equipment

    CN116811915A

  • Driving style identification method and system based on electroencephalogram data

    CN116861275A

  • Driver emotional state detection method and device based on electroencephalogram data

    CN117204856A

  • Cross-session brainprint recognition method based on tensorized spatial-frequency attention network (TSFAN) with domain adaptation

    US20240282439A1