The present invention provides an automatic
epilepsy signal recognition system and method based on multimodal
information fusion, wherein the electroencephalogram acquisition unit is used to acquire the EEG
signal of the patient in real time; the video acquisition unit is used to synchronously acquire the monitoring video
signal of the patient; the data preprocessing unit performs
noise reduction
processing according to the EEG signal and the monitoring video signal, and adds the EEG signal and the monitoring video signal to the
epileptic seizure start and end timestamps for
time series calibration to obtain epileptic electroencephalogram data and epileptic video data; the multi-scale convolutional network combines the
time domain,
frequency domain, time-
frequency domain and time-space domain features of the epileptic electroencephalogram data through a
deep learning feature extraction method to obtain epileptic electroencephalogram multi-feature vectors; the multi-level feature
pyramid network processes the epileptic video data through a spatial
pyramid pooling method to obtain epileptic video multi-feature vectors scale features; the human
optical flow estimation module obtains the
epilepsy video
optical flow vector by capturing the
epilepsy video data sequence; the multimodal
feature fusion module adopts the multi-layer
perceptron mapping method to fuse the epilepsy EEG multi-
feature vector, the epilepsy video multi-scale features and the video
optical flow vector to obtain the epilepsy fusion
feature data; the epilepsy
signal classification module identifies the epilepsy fusion
feature data through a bidirectional long short-
term memory network with a multiple attention mechanism to obtain the epilepsy signal category; the epilepsy data automatic recognition model optimization module adopts the
cross entropy loss function to
train the epilepsy signal automatic recognition model; the present invention makes full use of various data in the patient's epilepsy detection, adopts multimodal information of different dimensions to capture the
spatiotemporal correlation of epilepsy signals, and improves the robustness of the
system.