Low-speed eye movement signal detection method based on electro-oculogram signals and electroencephalogram signals

Abstract

The invention relates to a low-speed eye movement signal detection method based on an electro-oculogram signal and an electroencephalogram signal, which comprises the following steps: acquiring a horizontal electro-oculogram signal and an electroencephalogram signal in real time, and synchronously acquiring a current window electro-oculogram sample and a current window electroencephalogram sample by utilizing a sliding window with preset time length and step length, the method comprises the following steps: inputting a current electro-oculogram signal feature map and a current electroencephalogram signal feature map into a pre-trained parallel multi-modal one-dimensional convolutional neural network, outputting the current electro-oculogram signal feature map and the current electroencephalogram signal feature map after convolution, normalization and pooling in a first feature extraction module and a second feature extraction module which are parallel and identical, and inputting the current electro-oculogram signal feature map and the current electroencephalogram signal feature map into a feature fusion module for fusion to obtain a fused feature map; and inputting the fused feature map into a classification module, carrying out linear weighting through a preset number of hidden layers, obtaining a weighted feature map, inputting the weighted feature map into an output layer, and judging whether a slow-speed eye movement signal exists in the current window moment or not by utilizing Softmax function classification. According to the invention, two different modes are utilized, and simple and convenient signals are extracted to realize accurate detection of slow eye movement.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary field of biomedical engineering and machine learning, and in particular to a method, device and application for detecting slow eye movement signals based on electrooculogram (EOG) signals and electroencephalogram (EEG) signals. Background Art

[0002] Fatigue driving is one of the major causes of traffic accidents. Under monotonous driving conditions, drivers often ignore early signs of drowsiness and fall asleep unconsciously. Once a driver falls asleep, it is very likely that a car accident will occur due to untimely response to external emergencies. In order to prevent serious traffic accidents caused by drowsiness while driving, it is necessary to accurately detect the driver's drowsiness and provide timely warnings.

[0003] It is well known that physiological signals can directly reflect the mechanism of fatigue. According to research in the field of sleep, the attenuation of EEG alpha waves is the most reliable physiological indicator of the sleep onset period (SOP). Secondly, for horizontal electrooculogram (HEOG) signals, slow eye movements (SEMs) will occur during the SOP process, which is also an important physiological indicator of the onset of sleep. Studies have shown that SEMs are correlated with electroencephalogram (EEG) signals. In the frequency range of 1-14 Hz, SEMs are negatively correlated with EEG power, while in the frequency range of 15-30 Hz, SEMs are positively correlated with EEG power. However, compared with weak EEG signals, HEOG signals have a higher signal-to-noise ratio and are more suitable for actual environments.

[0004] Early studies used only a single modality of HEOG signals to detect SEMs, but recent studies have combined EEG signals to improve the detection accuracy of SEMs. An existing technology is to use a bimodal long short-term memory network (Bimodal-LSTM) to detect SEMs. This technology uses fast Fourier transform (FFT) and wavelet transform to extract EEG signal and HEOG signal features respectively. Subsequently, a bimodal long short-term memory network is used to process these features and perform classification tasks. However, the feature extraction of this technology still relies on manual design and cannot automatically learn effective features; but the features extracted by manual design are not accurate enough, resulting in poor detection accuracy using this feature, and low feature extraction efficiency and detection efficiency.

[0005] In summary, most of the existing slow eye movement detection methods only use single-modal information, which has certain information limitations; and the manually designed feature extraction methods are inaccurate, resulting in low detection efficiency and poor detection accuracy. Summary of the invention

[0006] To this end, the technical problem to be solved by the present invention is to overcome the problems of poor detection accuracy and low efficiency in the prior art of manually designing feature extraction methods to extract features of single-modal information for slow eye movement signal detection.

[0007] In order to solve the above technical problems, the present invention provides a method for detecting slow eye movement signals based on electrooculographic signals and electroencephalographic signals, comprising:

[0008] Collect a channel of horizontal electrooculogram signal and a channel of electroencephalogram signal in real time, slide on the channel of horizontal electrooculogram signal and the channel of electroencephalogram signal synchronously using a sliding window with preset time length and step length, obtain the current window electrooculogram sample and the current window electroencephalogram sample of each sliding, and input them into a pre-trained parallel multimodal one-dimensional convolutional neural network;

[0009] Input the current window electrooculogram sample and the current window electroencephalogram sample into the first feature extraction module and the second feature extraction module in parallel and the same, respectively, and after convolution, normalization and pooling, output the current electrooculogram signal feature map and the current electroencephalogram signal feature map;

[0010] Input the current electrooculogram signal feature map and the current electroencephalogram signal feature map into the feature fusion module for fusion to obtain a fusion feature map;

[0011] The fused feature map is input into the classification module, and linearly weighted after passing through a preset number of hidden layers to obtain a weighted feature map;

[0012] The weighted feature map is input into the output layer and classified using the Softmax function. According to the classification result, it is determined whether there is a slow eye movement signal in the current window moment represented by the current window electrooculogram sample and the current window electroencephalogram sample.

[0013] In one embodiment of the present invention, the first feature extraction module and the second feature extraction module both include a first one-dimensional convolution layer, a first batch normalization layer, a maximum pooling layer, a second one-dimensional convolution layer, a second batch normalization layer and a global average pooling layer; using the feature extraction module to extract a current signal feature map based on the current window sample, including:

[0014] The current window sample is used as the input feature map, input into the first one-dimensional convolution layer of the first feature extraction module for convolution operation, and outputs a one-dimensional convolution feature map, which is expressed as: i represents the index of the convolution kernel, the number of convolution kernels is I, 1≤i≤I; Represents the jth element in the l+1th layer of the one-dimensional convolutional feature map; represents the i-th convolution kernel of the l-th layer of the one-dimensional convolution feature map, and the symbol * represents the dot product. Represents the jth local area corresponding to the jth element of the lth layer of the one-dimensional convolutional feature map, and the local area is the area covered when the jth element of the convolution kernel moves on the lth layer of the one-dimensional convolutional feature map with a preset step size S; is the bias of the i-th convolution kernel in the l-th layer of the one-dimensional convolution feature map; when the length of the convolution kernel is K, the preset step size is S, and the padding size is P, the dimension of the one-dimensional convolution feature map is J, which is expressed as: 1≤j≤J;

[0015] The one-dimensional convolution feature map is input into the first batch of normalization layers. Based on the batch mean and batch variance, the one-dimensional convolution feature map is standardized, and then scaled and offset to obtain the normalized feature map, which is expressed as: γ represents the scale parameter, β represents the displacement parameter; Represents the standardized representation of the n-th one-dimensional convolutional feature map in the batch, expressed as B represents the preset batch size, m represents the number of one-dimensional convolutional feature maps in the preset batch size, 1≤n≤m, ε is a preset positive number; μ B represents the estimated mean value, expressed as represents the variance, and the expression is

[0016] The normalized feature map is input into the activation function layer, and the normalized feature map is activated using the ReLU activation function to obtain the activated feature map, which is expressed as: Represents the normalized feature map Activation feature map after activation; f represents the ReLU activation function;

[0017] The activation feature map is input into the maximum pooling layer, and the output feature dimension is reduced by downsampling, and the maximum pooling feature map is output, which is expressed as: t represents the position of the maximum pooling window on the input feature map, t∈[(j-1)w+1,jw], w represents the width of the pooling window;

[0018] The feature map of the maximum pooling layer is input into the second one-dimensional convolutional layer for convolution, and then input into the second batch normalization layer for activation through the ReLU activation function. The activated feature map is input into the global average pooling layer for pooling, and the current signal feature map is output.

[0019] In one embodiment of the present invention, the current electrooculogram signal feature map and the current electroencephalogram signal feature map are input into a feature fusion module for fusion to obtain a fused feature map, including splicing the current electrooculogram signal feature map and the current electroencephalogram signal feature map to obtain a fused feature map; the length of the fused feature map is equal to the sum of the lengths of the current electrooculogram signal feature map and the current electroencephalogram signal feature map.

[0020] In one embodiment of the present invention, the weighted feature map is input into the output layer, and the Softmax function is used for classification, and according to the classification result, it is determined whether there is a slow eye movement signal in the current window moment represented by the current window electrooculogram sample and the current window electroencephalogram sample, including:

[0021] If the classification value obtained by the Softmax function is 1, it is determined that there is a slow eye movement signal in the current window moment represented by the current window electrooculogram sample and the current window electroencephalogram sample;

[0022] If the classification value obtained by the Softmax function is 0, it is determined that there is no slow eye movement signal in the current window moment represented by the current window electrooculogram sample and the current window electroencephalogram sample.

[0023] In one embodiment of the present invention, the training process of the parallel multimodal one-dimensional convolutional neural network includes:

[0024] Synchronously collect one channel of eye level signal and one channel of EEG signal to obtain two channels of data;

[0025] Based on the slow eye movement waveform of one channel of electrooculographic horizontal signal, the two channels of data are marked as slow eye movement segments and non-slow eye movement segments;

[0026] Using a sliding window with preset duration and step length, the slow eye movement segments of one channel of eye level signal and one channel of EEG signal are synchronously slid to obtain a positive sample data set containing two-channel data window samples of slow eye movement;

[0027] Using a sliding window with preset duration and step length, synchronously slide the non-slow eye movement segments of one channel of eye level signal and one channel of EEG signal to obtain a negative sample data set of two-channel data window samples that do not contain slow eye movement;

[0028] The positive sample data set and the negative sample data set are used as training sets, and each pair of two-guide data window samples in the training set are used as input and respectively input into the two parallel feature extraction modules in the parallel multimodal one-dimensional convolutional neural network. The cross entropy loss function is used to train the parallel multimodal one-dimensional convolutional neural network to obtain the trained parallel multimodal one-dimensional convolutional neural network.

[0029] In one embodiment of the present invention, the cross entropy loss function is expressed as:

[0030]

[0031] Among them, t i represents the standard target label; P k Represents the predicted output result, the expression is zk Represents the output logarithm of the k-th neuron in the output layer, 1≤k≤J, and J represents the dimension of the one-dimensional convolutional feature map.

[0032] In one embodiment of the present invention, after obtaining a pre-trained parallel multimodal one-dimensional convolutional neural network, the Adam optimization method is used to perform network optimization, including:

[0033] Initialize learning speed, EPOCH and validation set;

[0034] The cross entropy loss function value is transmitted back using a one-dimensional convolutional neural network back propagation algorithm, the mean and uncentered variance are calculated using the Adam optimization method, and the learning rate of each parameter is adaptively adjusted; the mean is the first-order moment estimate of the gradient; the uncentered variance is the second-order moment estimate of the gradient;

[0035] The preset early stopping strategy monitors the classification accuracy of the validation set until the classification accuracy of the validation set no longer increases after a preset number of EPOCHs, and then stops training to optimize the pre-trained parallel multimodal one-dimensional convolutional neural network.

[0036] In one embodiment of the present invention, before using the positive sample data set and the negative sample data set as training sets, it also includes: downsampling all data samples in the positive sample data set and the negative sample data set respectively to generate data samples of preset length as training set data.

[0037] The embodiment of the present invention further provides a slow eye movement signal detection device based on electrooculographic signals and electroencephalographic signals, comprising:

[0038] An electrooculogram signal acquisition device, used for acquiring a horizontal electrooculogram signal;

[0039] An electroencephalogram signal acquisition device, used for acquiring an electroencephalogram signal;

[0040] A host computer is connected to the EOG signal acquisition device and the EEG signal acquisition device for acquiring a horizontal EOG signal and an EEG signal, and is used to execute the slow eye movement signal detection method based on EOG signal and EEG signal as described in any one of claims 1 to 8, to determine whether there is a slow eye movement signal in the current window moment represented by the EOG sample in the current window and the EEG sample in the current window, and to obtain a determination result;

[0041] The display device is connected to the host computer for displaying the judgment result.

[0042] The embodiment of the present invention further provides an application of the above-mentioned slow eye movement signal detection method based on electrooculogram signals and electroencephalogram signals in the field of drowsiness state detection.

[0043] The above technical solution of the present invention has the following advantages compared with the prior art:

[0044] The slow eye movement signal detection method based on EOG signals and EEG signals of the present invention utilizes the correlation and complementarity between EOG signals and EEG signals, utilizes EEG signals to assist EOG signals to detect slow eye movements, and constructs a parallel multimodal one-dimensional convolutional neural network; the input of the parallel multimodal one-dimensional convolutional neural network of the present invention includes two parallel and identical feature extraction modules for synchronously extracting feature maps of EOG signals and EEG signals, and does not need to manually design features based on professional experience. Instead, a parallel multimodal one-dimensional convolutional neural network is used to directly extract feature maps from EOG signals and EEG signals in an end-to-end manner. The invention automatically learns features and performs classification, thereby improving detection efficiency. The invention automatically extracts rich and complementary feature maps based on two modal information of electrooculogram signals and electroencephalogram signals, performs splicing and fusion, and uses a classification module to perform classification, thereby determining whether there is slow eye movement in the current window. The acquisition of one-channel horizontal electrooculogram signals and one-channel electroencephalogram data is relatively simple and easy to collect. The parallel multimodal one-dimensional convolutional neural network of the invention is used to detect slow eye movement, and the data of two modalities including electrooculogram signals and electroencephalogram signals are input, so that the data acquisition is simple, the feature extraction is accurate and efficient, and the accuracy of slow eye movement detection is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein

[0046] Figure 1 It is a flowchart of the steps of the slow eye movement signal detection method based on the electrooculogram signal and the electroencephalogram signal provided by the present invention;

[0047] Figure 2 It is a structural diagram of a parallel multimodal one-dimensional convolutional neural network of a slow eye movement signal detection method based on electrooculogram signals and electroencephalogram signals provided by the present invention;

[0048] Figure 3 (a) a schematic diagram of the positions of the two guide eye electric motors provided by the present invention; Figure 3 (b) a schematic diagram of the position of an electroencephalogram provided by the present invention;

[0049] Figure 4 It is a schematic diagram of the current window electrooculogram sample and the current window electroencephalogram sample provided by the present invention. DETAILED DESCRIPTION

[0050] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0051] In the analysis and processing of bioelectric signals, digital signal processing methods and machine learning methods have been widely used. Recently, building a multimodal deep learning model can more effectively realize feature fusion and improve the final classification accuracy. The deep learning method shows performance advantages over traditional machine learning methods. Therefore, the present invention is based on two modalities, electrooculogram signals and electroencephalogram signals, and uses a parallel multimodal one-dimensional convolutional neural network model to achieve the classification of slow eye movements.

[0052] Reference Figure 1 As shown, the flowchart of the method for detecting slow eye movement based on electrooculogram and electroencephalogram signals of the present invention includes the following specific steps:

[0053] S101: collecting a horizontal electrooculogram signal and an electroencephalogram signal in real time, sliding a sliding window with preset time length and step length on the horizontal electrooculogram signal and the electroencephalogram signal synchronously, obtaining the electrooculogram sample and the electroencephalogram sample of the current window of each sliding, and inputting them into a pre-trained parallel multimodal one-dimensional convolutional neural network;

[0054] S102: Input the current window electrooculogram sample and the current window electroencephalogram sample into the first feature extraction module and the second feature extraction module which are parallel and identical, respectively, and output the current electrooculogram signal feature map and the current electroencephalogram signal feature map after convolution, normalization and pooling;

[0055] S103: Inputting the current electrooculogram signal feature map and the current electroencephalogram signal feature map into a feature fusion module for fusion to obtain a fusion feature map;

[0056] S104: Input the fused feature map into the classification module, pass through a preset number of hidden layers, perform linear weighting, and obtain a weighted feature map;

[0057] S105: Input the weighted feature map into the output layer, perform classification using the Softmax function, and determine whether there is a slow eye movement signal in the current window moment represented by the current window electrooculogram sample and the current window electroencephalogram sample according to the classification result.

[0058] The invention discloses a method for detecting slow eye movement signals by fusing electrooculogram signals and electroencephalogram signals, which utilizes the features of electroencephalogram signals to assist in detecting slow eye movement, aiming at the correlation and complementarity between electrooculogram signals and electroencephalogram signals, and constructs a parallel multimodal one-dimensional convolutional neural network. The technology integrates two modal information of electrooculogram signals and electroencephalogram signals, and the neural network can automatically extract rich and complementary features from the signals of the two modalities by processing the electrooculogram signals and electroencephalogram signals in parallel, and automatically extracts features from the electrooculogram signals and electroencephalogram signals using two parallel feature extraction modules containing multiple convolutional layers, and fuses the one-dimensional electrooculogram signal feature map and the one-dimensional electroencephalogram signal feature map into a single fused one-dimensional feature map through a feature fusion module, and classifies the map using a classification module to determine whether the current window data has slow eye movement. The invention realizes simple data acquisition and high-precision detection by using a one-channel horizontal electrooculogram signal and a one-channel electroencephalogram signal to detect slow eye movement.

[0059] Specifically, in step S102, the current window electrooculogram sample and the current window electroencephalogram sample are respectively input into the first feature extraction module and the second feature extraction module to obtain the current electrooculogram signal feature map and the current electroencephalogram signal feature map; wherein the first feature extraction module and the second feature extraction module both include a first one-dimensional convolution layer, a first batch of normalization layers, a maximum pooling layer, a second one-dimensional convolution layer, a second batch of normalization layers and a global average pooling layer; using the feature extraction module to extract the current signal feature map based on the current window sample, including:

[0060] S102-1: The current window sample is used as an input feature map, input into the first one-dimensional convolution layer of the first feature extraction module for convolution operation, and a one-dimensional convolution feature map is output, which is expressed as: i represents the index of the convolution kernel, the number of convolution kernels is I, 1≤i≤I; Represents the jth element in the l+1th layer of the one-dimensional convolutional feature map; represents the i-th convolution kernel of the l-th layer of the one-dimensional convolution feature map, and the symbol * represents the dot product. Represents the jth local area corresponding to the jth element of the lth layer of the one-dimensional convolutional feature map, and the local area is the area covered when the jth element of the convolution kernel moves on the lth layer of the one-dimensional convolutional feature map with a preset step size S; is the bias of the i-th convolution kernel in the l-th layer of the one-dimensional convolution feature map; when the length of the convolution kernel is K, the preset step size is S, and the padding size is P, the dimension of the one-dimensional convolution feature map is J, which is expressed as: 1≤j≤J;

[0061] S102-2: Input the one-dimensional convolution feature map into the first batch of normalization layers, standardize the one-dimensional convolution feature map based on the batch mean and batch variance, and then scale and offset it to obtain the normalized feature map, which is expressed as: γ represents the scale parameter, β represents the displacement parameter; Represents the standardized representation of the n-th one-dimensional convolutional feature map in the batch, expressed as B represents the preset batch size, m represents the number of one-dimensional convolutional feature maps in the preset batch size, 1≤n≤m, ε is a preset positive number; μ B represents the estimated mean value, expressed as represents the variance, and the expression is

[0062] S102-3: Input the normalized feature map into the activation function layer, activate the normalized feature map using the ReLU activation function, and obtain the activated feature map, which is expressed as: Represents the normalized feature map Activation feature map after activation; f represents the ReLU activation function;

[0063] S102-4: Input the activated feature map into the maximum pooling layer, use downsampling to reduce the output feature dimension, and output the maximum pooling feature map, which is expressed as: t represents the position of the maximum pooling window on the input feature map, t∈[(j-1)w+1,jw], w represents the width of the pooling window;

[0064] S102-5: Input the feature map of the maximum pooling layer into the second one-dimensional convolutional layer for convolution, input it into the second batch normalization layer and then activate it through the ReLU activation function, input the activated feature map into the global average pooling layer for pooling, and output the current signal feature map.

[0065] Among them, the second one-dimensional convolution layer after the maximum pooling layer can expand the receptive field and enhance the feature learning ability. Its convolution operation is the same as the first one-dimensional convolution layer, but it uses convolution kernels of different numbers and lengths, and the activation function also uses ReLU; the second batch normalization processing layer performs processing similar to the first batch normalization processing layer before using the activation function on the output of the second one-dimensional convolution layer; the global average pooling layer is the last layer of the feature extraction module. The use of global average pooling can replace the traditional hidden layer in the convolutional neural network, reduce the number of parameters, and alleviate the overfitting problem.

[0066] In step S103, obtaining the fused feature map includes splicing the current electrooculogram signal feature map and the current electroencephalogram signal feature map to obtain the fused feature map; the length of the spliced ​​fused feature map is equal to the sum of the lengths of the current electrooculogram signal feature map and the current electroencephalogram signal feature map.

[0067] The training process of the parallel multimodal one-dimensional convolutional neural network includes:

[0068] Synchronously collect one channel of eye level signal and one channel of EEG signal to obtain two channels of data;

[0069] Based on the slow eye movement waveform of one channel of electrooculographic horizontal signal, the two channels of data are marked as slow eye movement segments and non-slow eye movement segments;

[0070] Using a sliding window with preset duration and step length, the slow eye movement segments of one channel of eye level signal and one channel of EEG signal are synchronously slid to obtain a positive sample data set containing two-channel data window samples of slow eye movement;

[0071] Using a sliding window with preset duration and step length, synchronously slide the non-slow eye movement segments of one channel of eye level signal and one channel of EEG signal to obtain a negative sample data set of two-channel data window samples that do not contain slow eye movement;

[0072] The positive sample data set and the negative sample data set are used as training sets, and each pair of two-guide data window samples in the training set are used as input and respectively input into the two parallel feature extraction modules in the parallel multimodal one-dimensional convolutional neural network. The cross entropy loss function is used to train the parallel multimodal one-dimensional convolutional neural network to obtain the trained parallel multimodal one-dimensional convolutional neural network.

[0073] Among them, the cross entropy loss function is expressed as: t i represents the standard target label, that is, the true value of whether the input feature map contains slow eye movement signals; P k Represents the predicted output result, the expression is z k Represents the output logarithm of the k-th neuron in the output layer, 1≤k≤J, and J represents the dimension of the one-dimensional convolutional feature map.

[0074] In this embodiment, after obtaining a pre-trained parallel multimodal one-dimensional convolutional neural network, the Adam optimization method is used to perform network optimization. The specific optimization steps include:

[0075] Initialize learning speed, EPOCH and validation set;

[0076] The cross entropy loss function value is transmitted back using a one-dimensional convolutional neural network back propagation algorithm, the mean and uncentered variance are calculated using the Adam optimization method, and the learning rate of each parameter is adaptively adjusted; the mean is the first-order moment estimate of the gradient; the uncentered variance is the second-order moment estimate of the gradient;

[0077] The preset early stopping strategy monitors the classification accuracy of the validation set until the classification accuracy of the validation set no longer increases after a preset number of EPOCHs, and then stops training to optimize the pre-trained parallel multimodal one-dimensional convolutional neural network.

[0078] Specifically, when training the parallel multimodal one-dimensional convolutional neural network, before using the positive sample data set and the negative sample data set as the training set, it also includes: downsampling all data samples in the positive sample data set and the negative sample data set respectively to generate data samples of a preset length as the training set data.

[0079] The method for detecting slow eye movement signals by fusing electrooculogram signals and electroencephalogram signals of the present invention integrates two modal information of electrooculogram signals and electroencephalogram signals according to the correlation and complementarity of electrooculogram signals and electroencephalogram signals, utilizes the features on the electroencephalogram signals to assist in detecting slow eye movement, and proposes a detection method based on a parallel multimodal one-dimensional convolutional neural network model. The present invention utilizes a parallel multimodal one-dimensional convolutional neural network to integrate two modal information of electrooculogram signals and electroencephalogram signals, and automatically learns features and performs classification directly from two modal data in an end-to-end manner, thereby improving detection efficiency; utilizes a parallel multimodal one-dimensional convolutional neural network to perform feature extraction and processing on electrooculogram signals and electroencephalogram signals in parallel, and fuses two one-dimensional feature maps of different modalities into a single one-dimensional fused feature map through a feature fusion module; utilizes a classification module to perform classification, thereby determining whether the driver is in a drowsy state; the present invention utilizes a horizontal electrooculogram signal and an electroencephalogram signal to detect the driver's drowsy state, and the data acquisition is simple and the detection accuracy is high.

[0080] Based on the above embodiments, in this embodiment, a slow eye movement detection device based on electrooculographic signals and electroencephalographic signals is further provided, including:

[0081] An electrooculogram signal acquisition device, used for acquiring a horizontal electrooculogram signal;

[0082] An electroencephalogram signal acquisition device, used for acquiring an electroencephalogram signal;

[0083] A host computer is connected to the EOG signal acquisition device and the EEG signal acquisition device for acquiring a horizontal EOG signal and an EEG signal, and performs the above-mentioned slow eye movement signal detection method based on EOG signal and EEG signal to determine whether there is a slow eye movement signal in the current window moment represented by the EOG sample in the current window and the EEG sample in the current window, and obtain a determination result;

[0084] The display device is connected to the host computer for displaying the judgment result.

[0085] Among them, the host computer is used to implement steps S101, S102, S103, S104 and S105 in the above-mentioned slow eye movement detection method based on electrooculogram signals and electroencephalogram signals. Therefore, its specific implementation method can refer to the description of the corresponding various partial embodiments and will not be repeated here.

[0086] An embodiment of the present invention also provides an application of the slow eye movement signal detection method based on EOG signals and EEG signals as described above in the field of drowsiness detection. In this embodiment, a simulated driving experiment is set up, and the experimental subjects are drivers with a habit of taking naps. A total of 20 drivers are recruited to participate in this experiment. The experiment is scheduled to be carried out 20 minutes before their usual nap time to ensure that the subjects conduct a two-hour simulated driving experiment in a sleepy state, and they are required to complete the Epworth Sleepiness Scale, with an ESS total score of 10.2±1.3, indicating a high degree of daytime sleepiness; one channel of EOG signals and one channel of EEG signals are collected from the driver in a simulated driving environment to detect the onset of the driver's drowsiness. Specifically, refer to Figure 2 As shown, using the pre-trained parallel multimodal one-dimensional convolutional neural network of the present invention, based on the current window electrooculogram samples and the current window electroencephalogram samples, slow eye movement signal detection is performed, and the specific steps include:

[0087] S201: Simulated driving experiment setting; In the simulated driving experiment environment, the driver sits in a real car without an engine, and a liquid crystal screen erected in front of the car can display virtual driving conditions; the driver can control the virtual car through the steering wheel to drive in the virtual driving conditions;

[0088] S202: using two electrodes disposed at the outer corners of the left and right eyes of the driver to collect a horizontal electrooculogram signal; using an electrode disposed behind the driver's head to collect a brain electroencephalogram signal;

[0089] Reference Figure 3 As shown in (a), two horizontal electrooculogram electrode signals Hr and Hl located at the outer corners of the left and right eyes are collected, and the traditional electrooculogram signal HEOG is obtained by subtracting the two electrode signals Hr and Hl;

[0090] Reference Figure 3 As shown in (b), a signal O2 from an electrode located behind the brain is collected to obtain an EEG signal O2;

[0091] The collected horizontal electrooculogram signal HEOG is subjected to noise reduction processing by using the median average method; the collected electroencephalogram signal O2 is subjected to noise reduction processing by using the median average method, and then the O2 signal is subjected to wavelet decomposition and reconstruction, and finally the O2 signal is subjected to smoothing processing;

[0092] S203: Based on the frequency and amplitude characteristics of the slow eye movement signal itself: (1) a slow sinusoidal excursion lasting more than 2 seconds in the frequency range of 0.2-0.6 Hz; (2) a signal amplitude between 20 and 200 μV; (3) binocular synchronization, with the left and right horizontal electrooculogram channels deflecting in opposite phases; (4) the wavelet generation interval between the right and left channels is within 300 ms; (5) there is no artifact in the HEOG signal, i.e., the difference between the left horizontal EOG channel and the right horizontal EOG channel; using the traditional visual inspection method in the electrooculogram, the slow eye movement segments and the non-slow eye movement segments are marked in the processed synchronized horizontal electrooculogram signal HEOG and the EEG signal O2;

[0093] S204: Sample set construction:

[0094] A sliding window with a fixed duration of 3 seconds is used, and a sliding step length of 0.1 seconds is used to slide on all slow eye movement segments, and the window data of each slide is obtained to form a positive window data sample set S containing a synchronous one-channel horizontal electrooculogram signal and one-channel electroencephalogram signal containing slow eye movement. + ;

[0095] A sliding window with a fixed duration of 3 seconds is used, and a sliding step length of 0.1 seconds is used to slide on all non-slow eye movement segments, and the window data of each slide is obtained to form a negative window data sample set S containing a synchronous one-channel horizontal electrooculogram and one-channel electroencephalogram signal without slow eye movement. - ;

[0096] Reference Figure 4 As shown in FIG. 1 , it is a schematic diagram of the current window electrooculogram sample and the current window electroencephalogram sample of the present invention; since the data sampling rate is 1000 Hz, S + and S - Each window data sample in is 1500 data points;

[0097] The original two types of window data sample sets are S + and S -Each sample in is downsampled, so that the original data sample is downsampled from the original 1500 data points to 500 points, which is used as the training data of the parallel multimodal one-dimensional convolutional neural network model. Therefore, the length of the one-dimensional original data of the parallel multimodal one-dimensional convolutional neural network input is 500;

[0098] S205: Construct a parallel multimodal one-dimensional convolutional neural network using the positive window data sample set S + and the negative class window data sample set S - In the example, samples with a window data length of 500 are trained to obtain a trained parallel multimodal one-dimensional convolutional neural network model M, which includes two parallel feature extraction modules containing multiple convolutional layers, a feature fusion module, and a classification module in sequence;

[0099] S205-1: The inputs of the first one-dimensional convolutional layers in the two parallel feature extraction modules are respectively the electrooculogram signal X EOG and EEG signal X EEG ; Among them, the electrooculogram signal X EOG and EEG signal X EEG All are time series with a data length of 500;

[0100] For electrooculogram signal X EOG For example, the first parallel feature extraction module is input to obtain the current electrooculogram signal feature map M EOG ,include:

[0101] After the first convolutional layer, a one-dimensional convolution operation is performed, which is expressed as:

[0102] In the formula, i is the index of the convolution kernel, the number of convolution kernels I is 150, and the value range of i is 1 to 150; represents the jth element of the feature map of the l+1th layer, represents the i-th convolution kernel of the l-th layer, and the symbol * represents the dot product between the convolution kernel and the local area. represents the jth local area of ​​the lth layer. The local area refers to the area covered by the convolution kernel when it moves on the feature map of the lth layer with a step size S of 1 in the jth step. is the bias of the i-th convolution kernel at layer l. When the length K of the convolution kernel is 50, the step size S is 1, the padding size P is 0, and the dimension of the feature map after one-dimensional convolution is J, it is calculated as follows: 1≤j≤J, that is, the dimension J of the feature map after the first one-dimensional convolution is 451;

[0103] The ReLU activation function is introduced to activate the output after one-dimensional convolution, limiting the input value to a non-negative range and having a linear property on the positive semi-axis. The expression is:

[0104] In the formula, Represents the output value after convolution operation, express The activation value obtained after the activation function;

[0105] The first batch normalization layer normalizes the batch input data before applying the activation function to the output of the first convolutional layer to speed up the convergence of the model training, making the model training process more stable and avoiding gradient explosion or gradient vanishing. Batch normalization works differently during training and testing. During training, batch normalization estimates the mean μ based on the input batch. B and variance The expression is:

[0106]

[0107]

[0108] B is the input mini-batch data sample set and the size of the mini-batch data sample set m is 150. The normalized expression of each instance is: In the formula, ε is a small positive number that prevents division by 0. In addition, a conversion step is added to restore the network's ability to represent the original input data. The expression is: Where γ and β are the scale and shift parameters. However, during testing, batch normalization calculates the mean and variance not for a particular batch, but for the entire dataset;

[0109] The maximum pooling layer reduces the output feature dimension of the convolutional layer by downsampling. For one-dimensional time data, the maximum pooling operation downsamples the input representation by taking the maximum value over a spatial window of a specific size. For each pooling step, the spatial window moves by the step size. The length c of the pooling size is 1, the width w is 2, and the pooling step size s is 1. The maximum pooling expression is: In the formula, Represents the output after the maximum pooling layer, max{} represents the maximum pooling operation, t represents the position range of the maximum pooling window on the input feature map, t∈[(j-1)w+1,jw], w represents the width of the pooling area;

[0110] The second convolutional layer is after the above-mentioned maximum pooling layer, which can expand the receptive field and enhance the feature learning ability. Its convolution operation is the same as the first convolutional layer, but uses convolution kernels of different numbers and lengths. The number of convolutions in the second layer is 100, the size is 30, the step size is 1, and the padding is 0. After the second layer of convolution, the dimension size of the feature map is 196, and the activation function also uses ReLU;

[0111] The second batch normalization layer performs similar processing as the first batch normalization layer before applying the activation function to the output of the second convolutional layer.

[0112] Global average pooling layer, in the last layer of the feature extraction module, can replace the traditional hidden layer in the convolutional neural network, reduce the number of parameters, and alleviate the overfitting problem.

[0113] Similarly, for the EEG signal X EEG For example, the above operations on the electrooculogram signal are performed to extract the current EEG signal feature map M EEG ;

[0114] S205-2: The one-dimensional electrooculogram signal feature map M from the first feature extraction module and the second feature extraction module is extracted. EOG and one-dimensional EEG signal feature map M EEG , which is fused into a single one-dimensional fusion feature map M through the feature fusion module fusion =[M EOG ,M EEG ] and sent to the input layer.

[0115] S205-3: Linearly weighting the input one-dimensional fusion feature map by two hidden layers with preset numbers of neurons of 128 and 50 respectively;

[0116] The output layer converts the output z of the hidden layer into k Use the Softmax function for classification, the expression is as follows:

[0117]

[0118] Where P k For z k The probability obtained after softmax() classification. Then use the cross entropy loss function to calculate the loss value, the expression is as follows:

[0119]

[0120] Where P k is the kth output prediction, t i is the corresponding target label, one-hot encoded;

[0121] S206: The parallel multimodal one-dimensional convolutional neural network structure in this embodiment sets the number of validation sets to 20% of the training set, the initial learning rate to 0.001, and uses the adaptive moment estimation optimization method to adjust the learning rate of each parameter by calculating the first-order moment estimation (i.e., mean) and second-order moment estimation (i.e., uncentered variance) of the gradient. The learning rate is adaptively adjusted according to different parameters to make the Adam optimizer more efficient and stable when training the deep learning model;

[0122] Set an early stopping strategy to monitor the classification accuracy of the validation set until the classification accuracy of the validation set stops increasing after 5 consecutive epochs, then stop training and save the trained parallel multimodal one-dimensional convolutional neural network M.

[0123] S207: Obtain a synchronized one-channel horizontal electrooculogram signal and one-channel electroencephalogram signal of the driver to be detected, slide on the synchronized one-channel horizontal electrooculogram test data and one-channel electroencephalogram signal test data of the driver to be detected with a sliding window with a fixed duration of 3 seconds and a sliding step length of 0.1 seconds, and obtain the current synchronized window data sample X for each sliding EOG and X EEG ; Based on the above trained parallel multimodal one-dimensional convolutional neural network M, the current synchronized window data sample X EOG and X EEG Make a prediction to determine whether the driver is currently in a drowsy state with eyes closed.

[0124] Specifically, when performing drowsiness detection, the slow eye movement signal detection device for fusing EOG signals and EEG signals to identify slow eye movement signals provided in an embodiment of the present invention also includes an early warning device, which is communicatively connected to the host computer and is used to work according to the detection results of the host computer on the synchronized horizontal EOG signals and EEG signal data samples; if a slow eye movement signal is detected in the current window data sample, it is considered that the driver to be detected is currently in a drowsy state, and the early warning device issues a warning to remind the driver not to fall asleep to ensure driving safety.

[0125] The method for detecting slow eye movement signals by fusing EOG signals and EEG signals of the present invention utilizes a parallel multimodal one-dimensional convolutional neural network to classify and predict data samples of the current window, utilizes the correlation and complementarity between EOG signals and EEG signals, utilizes features on EEG signals to assist in detecting slow eye movements, and constructs a parallel multimodal one-dimensional convolutional neural network; the technology integrates two modal information of EOG signals and EEG signals, and by processing EOG signals and EEG signals in parallel, the neural network can automatically extract rich and complementary features from the two types of signals, and when slow eye movement is detected in the data samples of the current window, it is considered that the driver to be detected is currently in a sleep onset state or a drowsy state, and the warning device is driven to issue an audible and visual warning to alert the driver, thereby helping to avoid road traffic safety problems caused by fatigue or drowsiness.

[0126] The slow eye movement signal detection method, device and application based on EOG and EEG signals of the present invention utilize the correlation and complementarity between EOG and EEG signals, utilize EEG signals to assist EOG signals to detect slow eye movements, and construct a parallel multimodal one-dimensional convolutional neural network; the input of the parallel multimodal one-dimensional convolutional neural network of the present invention includes two parallel and identical feature extraction modules for synchronously extracting feature maps of EOG and EEG signals, without manually designing features based on professional experience, but using a parallel multimodal one-dimensional convolutional neural network to directly extract feature maps from EOG and EEG signals in an end-to-end manner. The invention automatically learns features from the signal and classifies them, thereby improving the detection efficiency. The invention automatically extracts rich and complementary feature maps based on the two modal information of the electrooculogram signal and the electroencephalogram signal, performs splicing and fusion, and uses a classification module to classify them, thereby determining whether there is slow eye movement in the current window. The acquisition of a one-channel horizontal electrooculogram signal and a one-channel electroencephalogram data are both relatively simple and easy to collect. The parallel multimodal one-dimensional convolutional neural network of the invention is used to detect slow eye movement, and the data of the two modalities including the electrooculogram signal and the electroencephalogram signal are input, so that the data acquisition is simple, the feature extraction is accurate and efficient, and the accuracy of slow eye movement detection is improved.

[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0128] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0131] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A method for detecting slow eye movement signals based on electrooculogram signals and electroencephalogram signals, characterized in that: include: Collect a channel of horizontal electrooculogram signal and a channel of electroencephalogram signal in real time, slide on the channel of horizontal electrooculogram signal and the channel of electroencephalogram signal synchronously using a sliding window with preset time length and step length, obtain the current window electrooculogram sample and the current window electroencephalogram sample of each sliding, and input them into a pre-trained parallel multimodal one-dimensional convolutional neural network; Input the current window electrooculogram sample and the current window electroencephalogram sample into the first feature extraction module and the second feature extraction module in parallel and the same, respectively, and after convolution, normalization and pooling, output the current electrooculogram signal feature map and the current electroencephalogram signal feature map; Input the current electrooculogram signal feature map and the current electroencephalogram signal feature map into the feature fusion module for fusion to obtain a fusion feature map; The fused feature map is input into the classification module, and linearly weighted after passing through a preset number of hidden layers to obtain a weighted feature map; The weighted feature map is input into the output layer and classified using the Softmax function. According to the classification result, it is determined whether there is a slow eye movement signal in the current window moment represented by the current window electrooculogram sample and the current window electroencephalogram sample.

2. The method for detecting slow eye movement signals based on electrooculogram signals and electroencephalogram signals according to claim 1, characterized in that: The first feature extraction module and the second feature extraction module both include a first one-dimensional convolution layer, a first batch normalization layer, a maximum pooling layer, a second one-dimensional convolution layer, a second batch normalization layer and a global average pooling layer; The feature extraction module is used to extract the current signal feature map based on the current window sample, including: The current window sample is used as the input feature map, input into the first one-dimensional convolution layer of the first feature extraction module for convolution operation, and outputs a one-dimensional convolution feature map, which is expressed as: i represents the index of the convolution kernel, the number of convolution kernels is I, 1≤i≤I; Represents the jth element in the l+1th layer of the one-dimensional convolutional feature map; W i l represents the i-th convolution kernel of the l-th layer of the one-dimensional convolution feature map, and the symbol * represents the dot product. Represents the jth local area corresponding to the jth element of the lth layer of the one-dimensional convolutional feature map, and the local area is the area covered when the jth element of the convolution kernel moves on the lth layer of the one-dimensional convolutional feature map with a preset step size S; is the bias of the i-th convolution kernel in the l-th layer of the one-dimensional convolution feature map; when the length of the convolution kernel is K, the preset step size is S, and the padding size is P, the dimension of the one-dimensional convolution feature map is J, which is expressed as: The one-dimensional convolution feature map is input into the first batch of normalization layers. Based on the batch mean and batch variance, the one-dimensional convolution feature map is standardized, and then scaled and offset to obtain the normalized feature map, which is expressed as: γ represents the scale parameter, β represents the displacement parameter; Represents the standardized representation of the n-th one-dimensional convolutional feature map in the batch, expressed as B represents the preset batch size, m represents the number of one-dimensional convolutional feature maps in the preset batch size, 1≤n≤m, ε is a preset positive number; μ B represents the estimated mean value, expressed as represents the variance, and the expression is The normalized feature map is input into the activation function layer, and the normalized feature map is activated using the ReLU activation function to obtain the activated feature map, which is expressed as: Represents the normalized feature map Activation feature map after activation; f represents the ReLU activation function; The activation feature map is input into the maximum pooling layer, and the output feature dimension is reduced by downsampling, and the maximum pooling feature map is output, which is expressed as: t represents the position of the maximum pooling window on the input feature map, t∈[(j-1)w+1,jw], w represents the width of the pooling window; The feature map of the maximum pooling layer is input into the second one-dimensional convolutional layer for convolution, and then input into the second batch normalization layer for activation through the ReLU activation function. The activated feature map is input into the global average pooling layer for pooling, and the current signal feature map is output.

3. The method for detecting slow eye movement signals based on electrooculogram signals and electroencephalogram signals according to claim 1, characterized in that: The current electrooculogram signal feature map and the current electroencephalogram signal feature map are input into a feature fusion module for fusion to obtain a fused feature map, including splicing the current electrooculogram signal feature map and the current electroencephalogram signal feature map to obtain a fused feature map; the length of the fused feature map is equal to the sum of the lengths of the current electrooculogram signal feature map and the current electroencephalogram signal feature map.

4. The method for detecting slow eye movement signals based on electrooculogram signals and electroencephalogram signals according to claim 1, characterized in that: The weighted feature map is input into the output layer, and the Softmax function is used for classification. According to the classification result, it is determined whether there is a slow eye movement signal in the current window moment represented by the current window electrooculogram sample and the current window electroencephalogram sample, including: If the classification value obtained by the Softmax function is 1, it is determined that there is a slow eye movement signal in the current window moment represented by the current window electrooculogram sample and the current window electroencephalogram sample; If the classification value obtained by the Softmax function is 0, it is determined that there is no slow eye movement signal in the current window moment represented by the current window electrooculogram sample and the current window electroencephalogram sample.

5. The method for detecting slow eye movement signals based on electrooculogram signals and electroencephalogram signals according to claim 1, characterized in that: The training process of the parallel multimodal one-dimensional convolutional neural network includes: Synchronously collect one channel of eye level signal and one channel of EEG signal to obtain two channels of data; Based on the slow eye movement waveform of one channel of electrooculographic horizontal signal, the two channels of data are marked as slow eye movement segments and non-slow eye movement segments; Using a sliding window with preset duration and step length, the slow eye movement segments of one channel of eye level signal and one channel of EEG signal are synchronously slid to obtain a positive sample data set containing two-channel data window samples of slow eye movement; Using a sliding window with preset duration and step length, synchronously slide the non-slow eye movement segments of one channel of eye level signal and one channel of EEG signal to obtain a negative sample data set of two-channel data window samples that do not contain slow eye movement; The positive sample data set and the negative sample data set are used as training sets, and each pair of two-guide data window samples in the training set are used as input and respectively input into the two parallel feature extraction modules in the parallel multimodal one-dimensional convolutional neural network. The cross entropy loss function is used to train the parallel multimodal one-dimensional convolutional neural network to obtain the trained parallel multimodal one-dimensional convolutional neural network.

6. The method for detecting slow eye movement signals based on electrooculogram signals and electroencephalogram signals according to claim 5, characterized in that: The cross entropy loss function is expressed as: Among them, t i represents the standard target label; P k Represents the predicted output result, the expression is z k Represents the output logarithm of the k-th neuron in the output layer, 1≤k≤J, and J represents the dimension of the one-dimensional convolutional feature map.

7. The method for detecting slow eye movement signals based on electrooculogram signals and electroencephalogram signals according to claim 6, characterized in that: After obtaining the pre-trained parallel multimodal one-dimensional convolutional neural network, the Adam optimization method is used to optimize the network, including: Initialize learning speed, EPOCH and validation set; The cross entropy loss function value is transmitted back using a one-dimensional convolutional neural network back propagation algorithm, the mean and uncentered variance are calculated using the Adam optimization method, and the learning rate of each parameter is adaptively adjusted; the mean is the first-order moment estimate of the gradient; the uncentered variance is the second-order moment estimate of the gradient; The preset early stopping strategy monitors the classification accuracy of the validation set until the classification accuracy of the validation set no longer increases after a preset number of EPOCHs, and then stops training to optimize the pre-trained parallel multimodal one-dimensional convolutional neural network.

8. The method for detecting slow eye movement signals based on electrooculogram signals and electroencephalogram signals according to claim 5, characterized in that: Before using the positive sample data set and the negative sample data set as training sets, the method further includes: downsampling all data samples in the positive sample data set and the negative sample data set respectively to generate data samples of a preset length as training set data.

9. A slow eye movement signal detection device based on electrooculogram signals and electroencephalogram signals, characterized in that: include: An electrooculogram signal acquisition device, used for acquiring a horizontal electrooculogram signal; An electroencephalogram signal acquisition device, used for acquiring an electroencephalogram signal; A host computer is connected to the EOG signal acquisition device and the EEG signal acquisition device for acquiring a horizontal EOG signal and an EEG signal, and is used to execute the slow eye movement signal detection method based on EOG signal and EEG signal as described in any one of claims 1 to 8, to determine whether there is a slow eye movement signal in the current window moment represented by the EOG sample in the current window and the EEG sample in the current window, and to obtain a determination result; The display device is connected to the host computer for displaying the judgment result.

10. An application of the slow eye movement signal detection method based on EO signals and EEG signals as claimed in any one of claims 1 to 8 in the field of drowsiness detection.

Images