An automatic epileptic signal recognition system and method based on multimodal information fusion

Abstract

The present invention provides an automatic epilepsy signal recognition system and method based on multimodal information fusion, wherein the electroencephalogram acquisition unit is used to acquire the EEG signal of the patient in real time; the video acquisition unit is used to synchronously acquire the monitoring video signal of the patient; the data preprocessing unit performs noise reduction processing according to the EEG signal and the monitoring video signal, and adds the EEG signal and the monitoring video signal to the epileptic seizure start and end timestamps for time series calibration to obtain epileptic electroencephalogram data and epileptic video data; the multi-scale convolutional network combines the time domain, frequency domain, time-frequency domain and time-space domain features of the epileptic electroencephalogram data through a deep learning feature extraction method to obtain epileptic electroencephalogram multi-feature vectors; the multi-level feature pyramid network processes the epileptic video data through a spatial pyramid pooling method to obtain epileptic video multi-feature vectors scale features; the human optical flow estimation module obtains the epilepsy video optical flow vector by capturing the epilepsy video data sequence; the multimodal feature fusion module adopts the multi-layer perceptron mapping method to fuse the epilepsy EEG multi-feature vector, the epilepsy video multi-scale features and the video optical flow vector to obtain the epilepsy fusion feature data; the epilepsy signal classification module identifies the epilepsy fusion feature data through a bidirectional long short-term memory network with a multiple attention mechanism to obtain the epilepsy signal category; the epilepsy data automatic recognition model optimization module adopts the cross entropy loss function to train the epilepsy signal automatic recognition model; the present invention makes full use of various data in the patient's epilepsy detection, adopts multimodal information of different dimensions to capture the spatiotemporal correlation of epilepsy signals, and improves the robustness of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal intelligent medical signal processing, and in particular to an automatic epilepsy signal recognition system and method based on multimodal information fusion. Background Art

[0002] Epilepsy is a chronic neurological disease caused by sudden abnormal and hypersynchronous discharges of groups of brain neurons, leading to transient brain dysfunction. Its seizures are characterized by randomness, repetitiveness, and stereotypes, which impose a serious burden on the life and health of epilepsy patients. According to statistics from the World Health Organization, the number of epilepsy patients worldwide has exceeded 50 million, of which children and adolescents account for more than 60%. Epilepsy not only causes the sudden loss of control of the patient's physical and cognitive functions, but may also cause mental health problems such as anxiety or depression, and even in extreme cases may lead to sudden death from epilepsy. At present, accurate detection of epilepsy has become the focus of research in the field of neuroscience, aiming to provide more effective diagnostic and treatment methods to improve the quality of life of patients.

[0003] Timely and accurate detection of epilepsy is crucial for subsequent effective treatment. Electroencephalography (EEG) is a commonly used epilepsy diagnosis technology that provides a window into the function of the nervous system by monitoring the electrophysiological activity of brain neurons. However, the conventional EEG detection time is short and it is difficult to capture epileptic discharges or clinical seizures. In contrast, video EEG can perform long-term synchronous detection of the patient's EEG and clinical conditions, making it easier to capture the seizures of epileptic patients and record the clinical manifestations during the seizure. However, the video EEG examination lasts for a long time, usually including the process of wakefulness-sleep-wakefulness, generally lasting more than 24 hours. Therefore, it is necessary to visualize and analyze the large amount of data generated during long-term video EEG detection. At present, the diagnosis of epilepsy relies on professional doctors to observe the clinical manifestations in the video and the presence of epileptic discharges in the EEG to comprehensively judge whether the patient has an epileptic seizure. However, the analysis of epilepsy detection data requires manual feature extraction, which is cumbersome and time-consuming, and clinicians must have the skills to accurately interpret EEG signals to ensure that the correct diagnosis is made in a timely manner. Although the existing epilepsy detection algorithms have achieved certain results, there are still problems such as complex feature extraction process and insufficient information extraction. At the same time, there is a lack of research on automatic epilepsy recognition algorithms based on multimodal signals, and there are certain limitations in analyzing and studying only EEG signals. Therefore, exploring and developing a fast and effective automatic epilepsy signal recognition method based on multimodal information fusion is crucial for the diagnosis and control of epilepsy. Summary of the invention

[0004] In view of the difficulties in the prior art, the present invention provides an automatic epilepsy signal recognition method based on multimodal information fusion to address the shortcomings of the prior art, such as single data modality, complex traditional feature extraction process and low accuracy of epilepsy signal classification. The present invention makes full use of various data in epilepsy detection of patients, uses multimodal information of different dimensions to capture the spatiotemporal correlation of epilepsy signals, and improves the robustness of the system.

[0005] In order to solve the problems of the prior art, the present invention adopts the following technical solutions: an automatic epilepsy signal recognition system based on multimodal information fusion, the automatic epilepsy signal recognition system comprising a data acquisition unit, a data preprocessing unit, an automatic epilepsy signal recognition model and an automatic epilepsy signal recognition model optimization unit; the data acquisition unit is composed of an EEG acquisition module and a video acquisition module; the data preprocessing unit is composed of an EEG preprocessing module and a video preprocessing module; the automatic epilepsy signal recognition model comprises a multi-level feature extraction module, a human optical flow estimation module, a multimodal feature fusion module and an epilepsy signal classification module; the multi-level feature extraction module is composed of a multi-level feature pyramid network and a multi-scale convolutional network; wherein:

[0006] The electroencephalogram acquisition unit is used to acquire the patient's EEG signal in real time;

[0007] The video acquisition unit is used to synchronously acquire monitoring video signals of patients;

[0008] The EEG data preprocessing unit performs noise reduction processing according to the EEG signal and the monitoring video signal, and adds the EEG signal and the monitoring video signal to the epileptic seizure start and end timestamps for timing calibration to obtain epileptic EEG data and epileptic video data;

[0009] The multi-scale convolutional network combines the time domain, frequency domain, time-frequency domain and time-space domain features of epileptic EEG data through a deep learning feature extraction method to obtain epileptic EEG multi-feature vectors;

[0010] The multi-level feature pyramid network processes the epilepsy video data through a spatial pyramid pooling method to obtain multi-scale features of the epilepsy video;

[0011] The human body optical flow estimation module obtains the epilepsy video optical flow vector by capturing the epilepsy video data sequence;

[0012] The multimodal feature fusion module adopts a multi-layer perceptron mapping method to fuse epileptic EEG multi-feature vectors, epileptic video multi-scale features and video optical flow vectors to obtain epileptic fusion feature data;

[0013] The epilepsy classification module identifies epilepsy fusion feature data through a bidirectional long short-term memory network with a multiple attention mechanism to obtain an epilepsy signal category;

[0014] The epilepsy data automatic recognition model optimization module uses a cross entropy loss function to train the epilepsy signal automatic recognition model.

[0015] Furthermore, the multi-level feature pyramid network processes epilepsy video data by a spatial pyramid pooling method to obtain a multi-scale feature process of the epilepsy video; including:

[0016] The first epilepsy dynamic feature map is obtained by capturing the subtle texture and local structure information of epilepsy video data;

[0017] According to the top-down features, the epilepsy video data is captured according to the following formula to obtain the second epilepsy dynamic feature map:

[0018] P i =U(P i+1 )+C i

[0019] Where: P i is the feature map of the i-th level, C i is the convolutional feature map of the i-th level. The high-level feature map is converted to the same number of channels and resolution as the low-level feature map through the convolution operation. U represents the upsampling operation;

[0020] The multi-scale features of epilepsy video are obtained by enlarging the second epilepsy dynamic feature map and fusing it with the first epilepsy dynamic feature map.

[0021] Furthermore, the human body optical flow estimation module obtains the epilepsy video optical flow vector process by capturing the epilepsy video data sequence; comprising:

[0022] The human optical flow estimation module obtains an epileptic video data sequence by characterizing the activity level of the region of interest;

[0023] The total optical flow value calculated for each frame of the epilepsy video data sequence was calculated using the following formula;

[0024]

[0025] Where: J(n) is the total optical flow value of the nth frame, J(n) measures the patient's motion intensity in the region of interest in the video frame, the width of the region of interest is W, the height is H, and the unit is pixel; V x (x, y, n) and V y (x, y, n) is the optical flow value along the X-axis and Y-axis at the pixel with coordinates (x, y), and δ(n) is the noise value;

[0026] RAFT is used to calculate the optical flow vector field V in the total optical flow value calculated for each frame. x (x, y, n) and V y (x, y, n) obtains the epileptic video optical flow vector.

[0027] Furthermore, the multimodal feature fusion module adopts a multi-layer perceptron mapping method to fuse epileptic EEG multi-feature vectors, epileptic video multi-scale features and video optical flow vectors to obtain epileptic fusion feature data; including:

[0028] The embedded method is used to fuse the features of epileptic EEG multi-feature vectors, epileptic video multi-scale features and video optical flow vectors into a unified feature space;

[0029] The features of epileptic EEG multi-feature vector, epileptic video multi-scale features and video optical flow vector are mapped into a shared low-dimensional space through an encoder;

[0030] The epileptic EEG multi-feature vector, epileptic video multi-scale features and video optical flow vector are reconstructed back to the original space through the decoder;

[0031] Modal weights are introduced to determine the importance of epileptic EEG multi-feature vectors, epileptic video multi-scale features and video optical flow vectors in feature fusion by training the weight learning network.

[0032] Through the nonlinear mapping function, the epileptic EEG multi-feature vectors, the epileptic video multi-scale features and the video optical flow vectors are mapped according to the following formula to construct the relationship between them in the fused feature space;

[0033]

[0034] Among them: F fused is the fused feature vector, N is the number of feature vectors, F i are the original eigenvectors from different modes, α i It is the weighting coefficient of different features, which is used to control the contribution of different modal features in fusion;

[0035] The multi-layer perceptron is constructed through the following formula for mapping relationship:

[0036] Y=σ(W3·σ(W2·σ(W1·F fused +b1)+b2)+b3)

[0037] Where: Y is the output of the multilayer perceptron, W1, W2, W3 are the weight matrices of each hidden layer, b1, b2, b3 are the bias vectors of each hidden layer, σ is the activation function, and this module uses the ReLU function for activation.

[0038] The present invention can also be implemented by the following technical solutions:

[0039] A method for automatic identification of epilepsy signals based on multimodal information fusion comprises the following steps:

[0040] The patient's EEG signal is collected in real time through the EEG acquisition unit;

[0041] Synchronously collect monitoring video signals of patients through a video acquisition unit;

[0042] Noise reduction processing is performed on the EEG signal and the monitoring video signal, and the EEG signal and the monitoring video signal are added to the epileptic seizure start and end timestamps for time series calibration to obtain epileptic EEG data and epileptic video data;

[0043] Through deep learning feature extraction method, the time domain, frequency domain, time-frequency domain and time-space domain features of epileptic EEG data are combined to obtain epileptic EEG multi-feature vectors;

[0044] The multi-scale features of epilepsy video are obtained by processing epilepsy video data using the spatial pyramid pooling method.

[0045] Obtain epileptic video optical flow vector by capturing epileptic video data sequence;

[0046] The multi-layer perceptron mapping method is used to fuse the epileptic EEG multi-feature vectors, the epileptic video multi-scale features and the video optical flow vector to obtain the epileptic fusion feature data;

[0047] The epilepsy signal category is obtained by identifying the epilepsy fusion feature data through a bidirectional long short-term memory network with multiple attention mechanisms;

[0048] The following cross entropy loss function is used to train the epilepsy signal automatic recognition model:

[0049]

[0050] in: represents the cross entropy loss function, x (i) represents the true label of sample i, represents the predicted label of sample i, and n represents the total number of samples.

[0051] Furthermore, the epilepsy video data is processed by a spatial pyramid pooling method to obtain a multi-scale feature process of the epilepsy video; including:

[0052] The first epilepsy dynamic feature map is obtained by capturing the subtle texture and local structure information of epilepsy video data;

[0053] According to the top-down features, the epilepsy video data is captured according to the following formula to obtain the second epilepsy dynamic feature map:

[0054] P i =U(P i+1 )+C i

[0055] Where: Pi is the feature map of the i-th level, C i is the convolutional feature map of the i-th level. The high-level feature map is converted to the same number of channels and resolution as the low-level feature map through the convolution operation. U represents the upsampling operation;

[0056] The multi-scale features of the epilepsy video are obtained by amplifying the second epilepsy dynamic feature map and fusing it with the first epilepsy dynamic feature map. Furthermore, the epilepsy video optical flow vector process is obtained by capturing the epilepsy video data sequence; including:

[0057] The epilepsy video data sequence is obtained by characterizing the activity level of the region of interest;

[0058] The total optical flow value calculated for each frame of the epilepsy video data sequence was calculated using the following formula;

[0059]

[0060] Where: J(n) is the total optical flow value of the nth frame, J(n) measures the patient's motion intensity in the region of interest in the video frame, the width of the region of interest is W, the height is H, and the unit is pixel; V x (x, y, n) and V y (x, y, n) is the optical flow value along the X-axis and Y-axis at the pixel with coordinates (x, y), and δ(n) is the noise value;

[0061] RAFT is used to calculate the optical flow vector field V in the total optical flow value calculated for each frame. x (x, y, n) and V y (x, y, n) obtains the epileptic video optical flow vector.

[0062] Furthermore, a multi-layer perceptron mapping method is used to fuse epileptic EEG multi-feature vectors, epileptic video multi-scale features and video optical flow vectors to obtain epileptic fusion feature data; including:

[0063] The embedded method is used to fuse the features of epileptic EEG multi-feature vectors, epileptic video multi-scale features and video optical flow vectors into a unified feature space;

[0064] The features of epileptic EEG multi-feature vector, epileptic video multi-scale features and video optical flow vector are mapped into a shared low-dimensional space through an encoder;

[0065] The epileptic EEG multi-feature vector, epileptic video multi-scale features and video optical flow vector are reconstructed back to the original space through the decoder;

[0066] Modal weights are introduced to determine the importance of epileptic EEG multi-feature vectors, epileptic video multi-scale features and video optical flow vectors in feature fusion by training the weight learning network.

[0067] Through the nonlinear mapping function, the epileptic EEG multi-feature vectors, the epileptic video multi-scale features and the video optical flow vectors are mapped according to the following formula to construct the relationship between them in the fused feature space;

[0068]

[0069] Among them: F fused is the fused feature vector, N is the number of feature vectors, F i are the original eigenvectors from different modes, α i It is the weighting coefficient of different features, which is used to control the contribution of different modal features in fusion;

[0070] The multi-layer perceptron is constructed through the following formula for mapping relationship:

[0071] Y=σ(W3·σ(W2·σ(W1·F fused +b1)+b2)+b3)

[0072] Where: Y is the output of the multilayer perceptron, W1, W2, W3 are the weight matrices of each hidden layer, b1, b2, b3 are the bias vectors of each hidden layer, σ is the activation function, and this module uses the ReLU function for activation.

[0073] Beneficial Effects

[0074] Compared with the traditional technical solution, the beneficial effects brought by the present invention are:

[0075] The present invention proposes an automatic epilepsy signal recognition method and system based on multimodal information fusion, which uses multimodal high-dimensional clinical data such as video and EEG to automatically recognize epilepsy signals. Compared with the traditional epilepsy detection method based on manual judgment by professional doctors, this method combines multiple modal information to obtain epileptic seizure characteristics more comprehensively. It has the advantages of fast computing speed and high prediction accuracy, and can effectively improve the accuracy and efficiency of epilepsy detection by clinicians. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 :Flowchart of the automatic identification method of epilepsy signals based on multimodal information fusion;

[0077] Figure 2 : Multi-level feature pyramid network framework diagram;

[0078] Figure 3 : Multi-scale convolutional network framework diagram. DETAILED DESCRIPTION

[0079] The following is combined with Figure 1 -Attached Figure 3The technical solution of the present invention is further described.

[0080] In view of the shortcomings of the prior art, such as single data modality, complex traditional feature extraction process and low epilepsy signal classification accuracy, the present invention provides an automatic epilepsy signal recognition method based on multimodal information fusion.

[0081] First, for the acquisition of multimodal video EEG signals, the present invention adopts 10-20 international standard EEG acquisition equipment to acquire the patient's EEG signals, and then a synchronously matched video acquisition system acquires the patient's clinical video signals.

[0082] Secondly, the collected multimodal signals are synchronously transmitted to the data preprocessing unit. After data cleaning and normalization, the quality of the multimodal data is improved, and a multimodal epilepsy recognition database is constructed to provide rich data for subsequent model training. Next, a multimodal information fusion epilepsy signal automatic recognition model is constructed to detect the preprocessed multimodal signals. The model fuses multi-scale information through a multi-level feature extraction module to identify different epilepsy signals at different scales. For video information, a multi-level feature pyramid network is used to obtain the multi-scale features of the target image, which can capture the texture and global structure in the image.

[0083] Then, the human optical flow estimation module is used to detect the movement of epileptic patients. By calculating the optical flow information between adjacent images, the patient's small movement displacement is quickly calculated and analyzed. For EEG information, a multi-scale convolutional network is used to decompose the input signal into scales of different resolutions using deep convolution in a channel manner, so that the model can extract features from EEG signals at different scales to further capture epilepsy characteristics in different fields of view.

[0084] Subsequently, a bidirectional long short-term memory network (BiLSTM) based on multiple attention mechanisms was used to classify epileptic EEG signals.

[0085] Finally, the data set is divided into training set, validation set and test set, and then input into the constructed epilepsy signal recognition model for training and testing, and finally the patient's epilepsy signal recognition results are obtained. This improves the robustness and interpretability of the system and provides a solid foundation for more effective epilepsy detection. Specifically, it includes:

[0086] An automatic epilepsy signal recognition system based on multimodal information fusion; the automatic epilepsy signal recognition system comprises 1. a data acquisition unit; 2. a data preprocessing unit; 3. an automatic epilepsy signal recognition model; 4. an automatic epilepsy signal recognition model optimization unit;

[0087] The data acquisition unit is composed of an EEG acquisition module and a video acquisition module; the data preprocessing unit is composed of an EEG preprocessing module and a video preprocessing module; the epilepsy signal automatic recognition model includes a multi-level feature extraction module, a human optical flow estimation module, a multimodal feature fusion module and an epilepsy signal classification module; the multi-level feature extraction module is composed of a multi-level feature pyramid network and a multi-scale convolutional network; wherein:

[0088] The electroencephalogram acquisition unit is used to acquire the patient's EEG signal in real time;

[0089] The video acquisition module of the video acquisition unit is used to synchronously acquire monitoring video signals of patients;

[0090] The EEG data preprocessing unit performs noise reduction processing according to the EEG signal and the monitoring video signal, and adds the EEG signal and the monitoring video signal to the epileptic seizure start and end timestamps for timing calibration to obtain epileptic EEG data and epileptic video data;

[0091] The multi-scale convolutional network combines the time domain, frequency domain, time-frequency domain and time-space domain features of epileptic EEG data through a deep learning feature extraction method to obtain epileptic EEG multi-feature vectors;

[0092] The multi-level feature pyramid network obtains multiple feature vectors of epilepsy video by using a spatial pyramid pooling method for epilepsy video data;

[0093] The human body optical flow estimation module obtains the epilepsy video optical flow vector by capturing the epilepsy video data sequence;

[0094] The multimodal feature fusion module adopts a multi-layer perceptron mapping method to fuse epileptic EEG multi-feature vectors, epileptic video multi-feature vectors and video optical flow vectors to obtain epileptic fusion feature data;

[0095] The epilepsy judgment module identifies epilepsy fusion feature data through a bidirectional long short-term memory network with a multiple attention mechanism to obtain an epilepsy signal category;

[0096] The epilepsy data automatic recognition model optimization module uses a cross entropy loss function to train the epilepsy signal automatic recognition model.

[0097] 1. Data acquisition unit

[0098] The data acquisition unit obtains multimodal epilepsy monitoring data from the hospital's video EEG monitoring center database, mainly including EEG and video monitoring data, as well as related clinical data, such as medical records, medical images, etc., to provide more comprehensive information to support epilepsy detection. In the data acquisition unit, the EEG acquisition module is responsible for real-time acquisition of the patient's EEG signal. The data acquisition unit includes a series of professional sensors and electronic components, which are designed to obtain electrophysiological data of brain neural activity in epilepsy patients. At the same time, the video acquisition module is responsible for synchronous acquisition of the patient's monitoring video signal. The module includes a high-resolution camera and storage device to capture the patient's facial expressions, eye movements, limb movements and other physiological characteristics through specific angles and fields of view. This helps to obtain the clinical symptoms of epilepsy patients and provide more comprehensive data support for comprehensive analysis. Then the data is accurately aligned to ensure the consistency of multimodal data. Through the collaborative work of the EEG acquisition module and the video acquisition module, high-quality EEG and video data are obtained, laying the foundation for subsequent data preprocessing and model training.

[0099] 2. Data preprocessing unit

[0100] The data preprocessing unit mainly consists of two parts, namely the EEG preprocessing module and the video preprocessing module.

[0101] The EEG preprocessing module first filters the EEG data, and uses a bandpass filter to filter out signals with frequencies higher than 35Hz and lower than 0.5Hz. Then, an artifact removal operation is performed to correct artifacts caused by problems such as resistance differences between electrodes. Artifacts are eliminated by calculating the signal differences at adjacent time points and performing differential operations to ensure the accuracy of the EEG signal. For noise removal in EEG signals, the present invention adopts a method of combining independent component analysis and wavelet transform, uses ICA to perform blind source separation on the original EEG signal, obtains a set of independent components, and uses wavelet transform to perform frequency domain analysis for each independent component, identifies and filters the noise components, and finally resynthesizes the noise-removed independent components into EEG signals. The combined method can effectively remove environmental noise and noise such as electromyography and eye movement, and improve the purity of EEG signals. By analyzing the static part of the signal, the size and trend of the baseline drift are estimated, and then the estimated drift is removed from the original signal. Finally, the EEG signal is segmented, and the continuous EEG signal is divided into time periods of 4 seconds, and these time periods are overlapped for 2 seconds by sliding windows. This not only solves the problem of imbalanced epileptic seizure data, but also facilitates the training and analysis of subsequent models.

[0102] The video preprocessing module first converts the video data into corresponding video frames, and uniformly scales them to 512*512 pixels, and then removes the video frames with poor effects to reduce the amount of data and improve the computational efficiency. In terms of image quality optimization, mean filtering is used for denoising to reduce random noise in the image and improve the recognizability of facial and expression features. Histogram equalization is then used to dynamically adjust the contrast and brightness to ensure that the image has good visibility under different lighting conditions. At the same time, edge enhancement filters are used for sharpening to highlight facial and expression details and improve image clarity. The present invention uses the Faster R-CNN model for patient detection and tracking, which includes a region proposal network for generating candidate regions and a fast convolutional network for classification and bounding box regression. The aspect ratio of the bounding box is set to 2, the input image size is set to 512*512, and the model is trained on labeled image data to ensure that the model is quickly and accurately positioned in the video to prevent other people's activities on the screen from interfering with subsequent epilepsy signal analysis.

[0103] Finally, synchronization markers are inserted into the EEG and video data, and the start and end timestamps of epileptic seizures are added for timing calibration. By aligning the two on the time axis, the consistency of multimodal information is ensured, making the final data set more reliable. The preprocessed data is then input into the automatic epilepsy signal recognition unit for multimodal information fusion.

[0104] 3. Automatic epilepsy signal recognition model The automatic epilepsy signal recognition model mainly detects clinical epilepsy signals by constructing an end-to-end multimodal information fusion epilepsy signal automatic recognition model. The model mainly includes a multi-level feature extraction module, a human optical flow estimation module, a multimodal feature fusion module and an epilepsy signal classification module. Among them, the multi-level feature extraction module includes a multi-level feature pyramid network and a multi-scale convolutional network, which can more comprehensively consider the spatiotemporal information in multimodal data and provide richer contextual information for the recognition of epilepsy signals.

[0105] Firstly, a multi-level feature extraction module is used to extract features from preprocessed data of different modalities. The module includes a multi-level feature pyramid network and a multi-scale convolutional network.

[0106] The multi-level feature pyramid network extracts multi-scale features of epilepsy patients in video signals at different scales through spatial pyramid pooling operations.

[0107] The multi-scale convolutional network automatically extracts deep features at different levels in EEG signals through deep learning feature extraction methods, and combines them with traditional features such as time domain, frequency domain, time-frequency domain and time-space domain. It can provide richer input features, thereby enhancing the deep learning model's ability to characterize epilepsy signals and further improving the accuracy of the epilepsy recognition model.

[0108] At the same time, for video signals, the model also extracts the patient's optical flow estimation features through the human optical flow estimation module to reflect the dynamic information of epileptic patients in the video.

[0109] Secondly, the multimodal feature fusion module is used to fuse the features of different modalities. First, the feature spaces of different modalities are aligned, and the features of different modalities are fused into a unified feature space using an embedded method. The features of different modalities are mapped into a shared low-dimensional space through the encoder network, and then reconstructed back into the original space through the decoder network. Then, modal weights are introduced, and the importance of each modality in feature fusion is determined by training the weight learning network. Finally, the features of each modality are mapped through nonlinear mapping functions such as multilayer perceptrons, so that they can better express their relationship in the fused feature space.

[0110] After feature fusion, the fused feature vector is fed into the epilepsy signal classification module, which aims to accurately classify and identify epilepsy signals. In order to further improve the performance and interpretability of the model, the present invention introduces a multi-attention mechanism, which enables doctors and researchers to better understand the decision-making process of the model, thereby improving the credibility and reliability of epilepsy signal identification.

[0111] (1) Multi-level feature extraction module

[0112] By integrating information of different scales of multimodal data, the module can effectively extract epilepsy signal features at different scales, thereby significantly improving the performance of the epilepsy recognition model. The multi-level feature extraction module includes a multi-level feature pyramid network for video signals and a multi-scale convolutional network for EEG signals.

[0113] The multi-level feature pyramid network can obtain multi-scale features of the target image at different scales, where: each pyramid level contains different levels of information. At the bottom of the pyramid, the network can capture the subtle texture and local structural information of the image. This level of feature is critical for identifying small changes in the movements of epileptic patients in the image. As the pyramid level increases, the network obtains more advanced semantic information and can understand a larger range of structures or objects in the image. The pyramid structure can fuse features at different levels, which helps to understand the information in the image more comprehensively. At the same time, the use of spatial pyramid pooling operations can effectively reduce the dimension of the feature map, retain key information, and enable the network to effectively capture the movement characteristics in epileptic seizures. Among them: the top-down feature propagation can be described as:

[0114] P i =U(P i+1 )+C i

[0115] Where: P i is the feature map of the i-th level, C i is the convolutional feature map of the i-th level. The high-level feature map is converted to the same number of channels and resolution as the low-level feature map through convolution operation. U represents the upsampling operation. By fusing the enlarged feature map with the feature map of the next level, richer semantic information can be obtained, such as Figure 2 shown.

[0116] The multi-scale convolutional network processes EEG signals to improve the model's ability to capture complex time-frequency domain features. The multi-scale convolutional network uses deep convolution in a channel-by-channel manner to decompose the input signal into scales of different resolutions, enabling the network to extract features from EEG signals at different scales. Deep convolution is used to independently extract features to automatically learn channel weights to further capture epilepsy features in different fields of view, such as Figure 3 shown.

[0117] (2) Human optical flow estimation module

[0118] After removing the interference of other people in the picture from the preprocessed video frames, optical flow estimation is performed on the continuous video frames. Optical flow estimation is a method of describing the movement of pixels in an image between consecutive frames, which can be used to capture the movement pattern of patients in video sequences. By calculating the optical flow information between adjacent frames, it is beneficial to quickly calculate and analyze the patient's small movement displacement. The present invention proposes to use a measurement method that characterizes the activity level of the region of interest to detect movement changes in epileptic seizures. The measure of the activity level of the region of interest is the total optical flow value calculated for each frame in the video sequence.

[0119]

[0120] Where: J(n) is the total optical flow value of the nth frame, J(n) measures the patient's motion intensity in the region of interest in the video frame, the width of the region of interest is W, the height is H, and the unit is pixel. x (x, y, n) and V y (x, y, n) is the optical flow value along the X-axis and Y-axis at the pixel with coordinates (x, y), and δ(n) is the noise value. The present invention uses the RAFT optical flow estimation algorithm to calculate the optical flow vector field V in the video frame. x (x, y, n) and V y (x, y, n), the movement of epileptic patients in video frames is detected by adding a threshold to the optical flow field, where the threshold is set to ensure that 10% of the pixels have an optical flow value greater than zero.

[0121] To facilitate subsequent processing, the present invention connects the optical flow field images generated by every five video frame sequences for processing, and then uses a multi-layer convolutional neural network (CNN) pre-trained on ImageNet for motion detection to extract the patient's epileptic seizure information.

[0122] (3) Multimodal feature fusion module

[0123] In multimodal feature fusion, the multi-scale deep learning features extracted by the multi-level feature extraction module, the traditional time-frequency domain features and the optical flow features extracted by the human optical flow estimation module are fused. First, the feature spaces of different modalities are aligned, and the features of different modalities are fused into a unified feature space using an embedded method. The features of different modalities are mapped to a shared low-dimensional space through the encoder network, and then reconstructed back to the original space through the decoder network. Then, modal weights are introduced, and the importance of each modality in feature fusion is determined by training the weight learning network. Finally, the features of each modality are mapped through nonlinear mapping functions such as multi-layer perceptrons, so that they can better express their relationship in the fused feature space. The feature fusion process can be described as:

[0124]

[0125] Among them: F fused is the fused feature vector, N is the number of feature vectors, F i are the original eigenvectors from different modes, α i is the weighting coefficient of different features, which is used to control the contribution of different modal features in fusion. The mapping process through the multi-layer perceptron can be described as:

[0126] Y=σ(W3·σ(W2·σ(W1·F fused +b1)+b2)+b3)

[0127] Where: Y is the output of the multi-layer perceptron, W1, W2, W3 are the weight matrices of each hidden layer, b1, b2, b3 are the bias vectors of each hidden layer, σ is the activation function, and this module uses the ReLU function for activation. The multi-layer perceptron network can further extract the high-order relationship between features, so that the fused features can better represent the correlation between different modalities in a higher-dimensional space. This helps the model to more accurately identify epileptic signals and improve the performance of the system.

[0128] (4) Epilepsy data classification module

[0129] The epilepsy data classification module uses a bidirectional long short-term memory network (BiLSTM) based on a multiple attention mechanism to identify epilepsy signals. The BiLSTM network can use two hidden layers, forward and reverse, for information processing. This unique structure allows the network to simultaneously process the forward and backward information of the time series signal, thereby providing a richer and more comprehensive representation of the information flow. The present invention helps to explain the decision-making process of the model by introducing an attention mechanism in the BiLSTM model. The attention mechanism assigns different weights to data from different channels, so that channels with larger weights can be focused in subsequent processing. In the task of epilepsy signal recognition, the attention mechanism assigns higher weights to important brain areas that cause epileptic seizures, thereby improving the accuracy and robustness of the model in the task of epilepsy signal recognition.

[0130] In the attention mechanism, a fully connected module and a nonlinear function are used to obtain a weighted matrix, in which each element is normalized to the interval [0, 1]. Then each element in the matrix is ​​calculated to obtain the mean of the matrix in a two-dimensional array. Specifically, let the input feature be x i , the fully connected module maps it to the attention weight β i , where W x and b x are the corresponding weights and biases, W a is the weight of the fully connected module.

[0131] β i =softmax(W a tanh(W x ·x i +b x ))

[0132] After adding weights, the corresponding signals are fed into the BiLSTM network for further processing. Such an attention mechanism allows the model to automatically learn the importance of each feature and adjust the input signal according to its importance to better capture the key information in the signal. The BiLSTM network is composed of several LSTM units connected together. Each LSTM unit has an input gate, a forget gate, and an output gate. It can effectively handle long-term dependencies and make full use of the timing information of sequence data. The input gate i t 、Forget gate t and output gate o t and memory unit c t As follows, τ represents the activation function, x t is the t-th time feature of the input sequence, h t-1 is the hidden state at the previous time, c t-1 is the memory unit of the previous time, W is the learned weight, and b is the bias vector. The calculation process of the LSTM unit is as follows:

[0133] i t =τ(W i ·(h t-1 , x t )+b i )

[0134] f t =τ(W f ·(h t-1 , x t )+b f )

[0135] o t =τ(W o ·(h t-1 , x t )+b o )

[0136]

[0137]

[0138] h t =o t ·tanh(c t )

[0139] y t =softmax(W y ·h t +b y )

[0140] Specifically, at each time t, the BiLSTM network will t and the hidden state h at the previous time t-1 To calculate the hidden state h at the current time t , and then pass the fully connected layer to transform the hidden state h t Mapped to the output space y t Through the back propagation algorithm and gradient descent, the network parameters are adjusted so that the model can more accurately identify epileptic signals. Finally, the BiLSTM network will output the prediction results of each time step, thereby realizing the recognition of epileptic signals.

[0141] 4. Epileptic signal automatic identification model optimization unit The model optimization unit uses the cross entropy loss function to constrain the overall training process to ensure that the model can effectively learn the data features during the training process and prevent overfitting. The cross entropy loss function is an indicator used to measure the difference between two probability distributions. In classification tasks, the output of the model usually represents the predicted probability of each category, and the cross entropy loss function can help the model optimize the probability prediction to make it as close to the actual category label as possible. represents the cross entropy loss function, x (i) represents the true label of sample i, represents the predicted label of sample i, n represents the total number of samples, then the loss function L ce It can be expressed as:

[0142]

[0143] The cross entropy loss function is applied to the output of the last layer classifier of the epilepsy signal recognition network, and the network predicts the probability distribution of the sample for each category. By calculating the cross entropy loss between the model's predicted label and the true label, the model parameters are optimized through back propagation, so that it can more accurately identify epilepsy signals. The model parameters are optimized by minimizing the cross entropy loss, so that the model can more accurately identify epilepsy signals. Secondly, a dynamic learning rate adjustment strategy is adopted to dynamically adjust the learning rate according to the performance of the model during the training process to accelerate the convergence speed of the model. Batch normalization technology is used to reduce the gradient problem during training. The model optimization unit effectively improves the performance and accuracy of the automatic epilepsy signal recognition system by comprehensively applying the above optimization techniques and methods.

[0144] Although the present invention has been described above, the present invention is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can make many modifications without departing from the purpose of the present invention, which are all within the protection of the present invention.

Claims

1. An automatic epilepsy signal recognition system based on multimodal information fusion, characterized by: The epilepsy signal automatic recognition system includes a data acquisition unit, a data preprocessing unit, an epilepsy signal automatic recognition model and an epilepsy signal automatic recognition model optimization unit; the data acquisition unit is composed of an EEG acquisition module and a video acquisition module; the data preprocessing unit is composed of an EEG preprocessing module and a video preprocessing module; the epilepsy signal automatic recognition model includes a multi-level feature extraction module, a human optical flow estimation module, a multimodal feature fusion module and an epilepsy signal classification module; the multi-level feature extraction module is composed of a multi-level feature pyramid network and a multi-scale convolutional network; wherein: The EEG acquisition module is used to collect the patient's EEG signal in real time; The video acquisition module is used to synchronously acquire monitoring video signals of patients; The data preprocessing unit performs noise reduction processing according to the EEG signal and the monitoring video signal, and adds the EEG signal and the monitoring video signal to the epileptic seizure start and end timestamps for timing calibration to obtain epileptic EEG data and epileptic video data; The multi-scale convolutional network extracts features of epileptic EEG data in time domain, frequency domain, time-frequency domain and time-space domain at different scales through a deep learning feature extraction method to obtain multiple feature vectors of epileptic EEG; The multi-level feature pyramid network processes epilepsy video data by a spatial pyramid pooling method to obtain multi-scale features of epilepsy video; The human body optical flow estimation module obtains the epilepsy video optical flow vector by capturing the epilepsy video data sequence; The multimodal feature fusion module adopts a multi-layer perceptron mapping method to fuse epileptic EEG multi-feature vectors, epileptic video multi-scale features and video optical flow vectors to obtain epileptic fusion feature data; The epilepsy signal classification module identifies epilepsy fusion feature data through a bidirectional long short-term memory network with a multiple attention mechanism to obtain an epilepsy signal category; The epilepsy signal automatic recognition model optimization unit uses a cross entropy loss function to train the epilepsy signal automatic recognition model.

2. The epilepsy signal automatic recognition system based on multimodal information fusion according to claim 1, characterized in that: The multi-level feature pyramid network processes epilepsy video data by a spatial pyramid pooling method to obtain multi-scale features of epilepsy video; including: The first epilepsy dynamic feature map is obtained by capturing the subtle texture and local structure information of epilepsy video data; According to the top-down features, the epilepsy video data is captured according to the following formula to obtain the second epilepsy dynamic feature map: P i =U(P i+1 )+C i Where: P i is the feature map of the i-th level, C i is the convolutional feature map of the i-th level. The high-level feature map is converted to the same number of channels and resolution as the low-level feature map through the convolution operation. U represents the upsampling operation; The multi-scale features of epilepsy video are obtained by enlarging the second epilepsy dynamic feature map and fusing it with the first epilepsy dynamic feature map.

3. The automatic epilepsy signal recognition system based on multimodal information fusion according to claim 1, characterized in that: The human body optical flow estimation module obtains the epilepsy video optical flow vector process by capturing the epilepsy video data sequence; comprising: The human optical flow estimation module obtains an epileptic video data sequence by using a method for characterizing the activity level of a region of interest; The total optical flow value calculated for each frame of the epilepsy video data sequence was calculated using the following formula; Where: J(n) is the total optical flow value of the nth frame, J(n) measures the patient's motion intensity in the region of interest in the video frame, the width of the region of interest is W, the height is H, and the unit is pixel; V x (x,y,n) and V y (x, y, n) is the optical flow value along the X-axis and Y-axis at the pixel with coordinates (x, y), and δ(n) is the noise value; RAFT is used to calculate the optical flow vector field V in the total optical flow value calculated for each frame. x (x,y,n) and V y (x,y,n) obtains the epileptic video optical flow vector.

4. The automatic epilepsy signal recognition system based on multimodal information fusion according to claim 1, characterized in that: The multimodal feature fusion module adopts a multi-layer perceptron mapping method to fuse epileptic EEG multi-feature vectors, epileptic video multi-scale features and video optical flow vectors to obtain epileptic fusion feature data; including: The embedded method is used to fuse the features of epileptic EEG multi-feature vectors, epileptic video multi-scale features and video optical flow vectors into a unified feature space; The features of epileptic EEG multi-feature vector, epileptic video multi-scale features and video optical flow vector are mapped into a shared low-dimensional space through an encoder; The epileptic EEG multi-feature vector, epileptic video multi-scale features and video optical flow vector are reconstructed back to the original space through the decoder; Modal weights are introduced to determine the importance of epileptic EEG multi-feature vectors, epileptic video multi-scale features and video optical flow vectors in feature fusion by training the weight learning network. The epileptic EEG multi-feature vector, epileptic video multi-scale feature and video optical flow vector are mapped according to the following formula through a nonlinear mapping function to construct the relationship between the fused epileptic EEG multi-feature vector, epileptic video multi-scale feature and video optical flow vector in the feature space: Among them: F fused is the fused feature vector, N is the number of feature vectors, F i are the original eigenvectors from different modes, α i It is the weighting coefficient of different features, which is used to control the contribution of different modal features in fusion; The multi-layer perceptron is constructed through the following formula for mapping relationship: Y=σ(W3.σ(W2.σ(W1.F fused +b1)+b2)+b3) Where: Y is the output of the multilayer perceptron, W1, W2, W3 are the weight matrices of each hidden layer, b1, b2, b3 are the bias vectors of each hidden layer, σ is the activation function, and this module uses the ReLU function for activation.

5. A method for automatically identifying epileptic signals by using the system of claim 1 with multimodal information fusion, characterized in that: The steps include: The patient's EEG signal is collected in real time through the EEG acquisition unit; Synchronously collect monitoring video signals of patients through a video acquisition unit; Noise reduction processing is performed on the EEG signal and the monitoring video signal, and the EEG signal and the monitoring video signal are added to the epileptic seizure start and end timestamps for time series calibration to obtain epileptic EEG data and epileptic video data; Through deep learning feature extraction method, epileptic EEG data are characterized in time domain, frequency domain, time-frequency domain and time-space domain at different scales to obtain epileptic EEG multi-feature vectors; The epilepsy video data is processed by spatial pyramid pooling method to obtain multi-scale features of epilepsy video; Obtain epileptic video optical flow vector by capturing epileptic video data sequence; The multi-layer perceptron mapping method is used to fuse the epileptic EEG multi-feature vectors, the epileptic video multi-scale features and the video optical flow vector to obtain the epileptic fusion feature data; The epilepsy signal category is obtained by identifying the epilepsy fusion feature data through a bidirectional long short-term memory network with multiple attention mechanisms; The following cross entropy loss function is used to train the epilepsy signal automatic recognition model: in: represents the cross entropy loss function, x (i) represents the true label of sample i, represents the predicted label of sample i, and n represents the total number of samples.

6. The method for automatically identifying epilepsy signals by multimodal information fusion according to claim 5, characterized in that: The epilepsy video data is processed by the spatial pyramid pooling method to obtain the multi-scale feature process of the epilepsy video; including: The first epilepsy dynamic feature map is obtained by capturing the subtle texture and local structure information of epilepsy video data; According to the top-down features, the epilepsy video data is captured according to the following formula to obtain the second epilepsy dynamic feature map: P i =U(P i+1 )+C i Where: P i is the feature map of the i-th level, C i is the convolutional feature map of the i-th level. The high-level feature map is converted to the same number of channels and resolution as the low-level feature map through the convolution operation. U represents the upsampling operation; The multi-scale features of epilepsy video are obtained by enlarging the second epilepsy dynamic feature map and fusing it with the first epilepsy dynamic feature map.

7. The method for automatically identifying epilepsy signals by using a system with multimodal information fusion according to claim 5, characterized in that: The epileptic video optical flow vector process is obtained by capturing the epileptic video data sequence; including: The epileptic video data sequence is obtained by using the method of characterizing the activity level of the region of interest; The total optical flow value calculated for each frame of the epilepsy video data sequence was calculated using the following formula; Where: J(n) is the total optical flow value of the nth frame, J(n) measures the patient's motion intensity in the region of interest in the video frame, the width of the region of interest is W, the height is H, and the unit is pixel; V x (x,y,n) and V y (x, y, n) is the optical flow value along the X-axis and Y-axis at the pixel with coordinates (x, y), and δ(n) is the noise value; RAFT is used to calculate the optical flow vector field V in the total optical flow value calculated for each frame. x (x,y,n) and V y (x,y,n) obtains the epileptic video optical flow vector.

8. The method for automatically identifying epilepsy signals by using a system with multimodal information fusion according to claim 5, characterized in that: The multi-layer perceptron mapping method is used to fuse the epilepsy EEG multi-feature vectors, the epilepsy video multi-scale features and the video optical flow vector to obtain the epilepsy fusion feature data process; including: The embedded method is used to fuse the features of epileptic EEG multi-feature vectors, epileptic video multi-scale features and video optical flow vectors into a unified feature space; The features of epileptic EEG multi-feature vector, epileptic video multi-scale features and video optical flow vector are mapped into a shared low-dimensional space through an encoder; The epileptic EEG multi-feature vector, epileptic video multi-scale features and video optical flow vector are reconstructed back to the original space through the decoder; Modal weights are introduced to determine the importance of epileptic EEG multi-feature vectors, epileptic video multi-scale features and video optical flow vectors in feature fusion by training the weight learning network. The epileptic EEG multi-feature vector, the epileptic video multi-scale feature and the video optical flow vector are mapped according to the following formula through a nonlinear mapping function to construct the relationship between the fused epileptic EEG multi-feature vector, the epileptic video multi-scale feature and the video optical flow vector in the feature space: Among them: F fused is the fused feature vector, N is the number of feature vectors, F i are the original eigenvectors from different modes, α i It is the weighting coefficient of different features, which is used to control the contribution of different modal features in fusion; The multi-layer perceptron is constructed through the following formula for mapping relationship: Y=σ(W3.σ(W2.σ(W1.F fused +b1)+b2)+b3) Where: Y is the output of the multilayer perceptron, W1, W2, W3 are the weight matrices of each hidden layer, b1, b2, b3 are the bias vectors of each hidden layer, σ is the activation function, and this module uses the ReLU function for activation.

Images