Short-term memory discrimination method based on multi-modal neuroimaging and deep learning

Abstract

The application discloses a short-term memory discrimination method based on multi-modal neuroimaging and deep learning, and comprises the following steps: S1, synchronizing an EEG-fMRI device to collect electroencephalogram signals and BOLD signals in a resting state and a memory task; S2, pre-processing the data collected in the step S1; S3, extracting feature data of the EEG and the fMRI through a convolutional neural network; S4, fusing the feature data of the EEG and the fMRI through position coding and attention mechanism weighting to generate a joint feature vector; and S5, classifying the joint feature vector based on an ELM algorithm to output a memory state discrimination result.The application has the beneficial effect that the fMRI and EEG data obtained through collection and fusion are combined with the CNN and ELM algorithms to realize high-precision memory state classification, the correlation analysis is performed with a behavioral result, the short-term memory characteristic signal is discriminated by using neuroimaging data, and the neural mechanism of memory coding and consolidation is analyzed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of brain science and artificial intelligence, in particular to a short-term memory discrimination method based on multi-modal neuroimaging and deep learning. BACKGROUND

[0002] At present, in the short-term memory discrimination, mainly through a single mode such as only EEG or fMRI, it is difficult to take into account the high space-time resolution, resulting in incomplete short-term memory feature extraction. The traditional analysis method cannot process multi-modal data in real time, and lacks dynamic tracking of memory encoding and consolidation stage, and the existing memory evaluation relies on behavioral indicators, lacking objective signal feedback at the level of neurophysiology. SUMMARY

[0003] The purpose of the present application is to provide a short-term memory discrimination method based on multi-modal neuroimaging and deep learning, which integrates EEG and fMRI data in real time, combines deep learning and ELM algorithm, realizes dynamic detection and output of short-term memory feature signals, and reveals the neural mechanism of memory encoding and consolidation stage.

[0004] The purpose of the present application is achieved by the following technical solutions:

[0005] The short-term memory discrimination method based on multi-modal neuroimaging and deep learning comprises the following steps:

[0006] S1, synchronizing EEG-fMRI equipment, collecting electroencephalogram signals and BOLD signals in resting state and memory task;

[0007] S2, pre-processing the data collected in S1 step;

[0008] S3, extracting feature data of EEG and fMRI through convolutional neural network;

[0009] S4, weighting and fusing the feature data of EEG and fMRI through position coding and attention mechanism to generate joint feature vector;

[0010] S5, classifying the joint feature vector based on ELM algorithm, and outputting the memory state discrimination result.

[0011] Further, the data preprocessing includes EEG signal preprocessing and fMRI signal preprocessing;

[0012] EEG signal preprocessing adopts adaptive filtering and independent component analysis to remove MRI gradient artifact and ECG artifact;

[0013] fMRI signal preprocessing is based on DPABI, SPM12 for head motion correction, spatial standardization and smoothing processing.

[0014] Further, in the S3 step, the feature data of the EEG includes: time-frequency domain features of theta waves and time-frequency domain features of gamma wave power.

[0015] The feature data of the fMRI includes BOLD signals of the hippocampus and the prefrontal lobe.

[0016] Further, the convolutional neural network is a depth separable convolutional neural network, and the separable convolutional neural network includes a depth convolution and a pointwise convolution.

[0017] The depth convolution performs independent convolution processing on each channel of the input.

[0018] The pointwise convolution fuses each channel by using a 1x1 convolution kernel.

[0019] Further, in the S4 step, the attention mechanism integrates the position coding information into the attention, and the obtained position coding information is used to quickly locate the position of interest.

[0020] The present application has the following advantages: by collecting and fusing the obtained fMRI and EEG data, combining the CNN and ELM algorithms to realize high-precision memory state classification, performing correlation analysis with the behavioral results, using the neural image data to judge the short-term memory characteristic signal, and analyzing the neural mechanism of memory encoding and consolidation. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The present application has the following advantages: by collecting and fusing the obtained fMRI and EEG data, combining the CNN and ELM algorithms to realize high-precision memory state classification, performing correlation analysis with the behavioral results, using the neural image data to judge the short-term memory characteristic signal, and analyzing the neural mechanism of memory encoding and consolidation.

[0022] Figure 2 The present application has the following advantages: by collecting and fusing the obtained fMRI and EEG data, combining the CNN and ELM algorithms to realize high-precision memory state classification, performing correlation analysis with the behavioral results, using the neural image data to judge the short-term memory characteristic signal, and analyzing the neural mechanism of memory encoding and consolidation. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0024] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0025] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0026] It should be noted that similar reference numerals and letters refer to similar items throughout the accompanying drawings, and once an item is defined in one drawing, it is not necessary to further define and explain it in subsequent drawings.

[0027] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship commonly understood by those skilled in the art, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0028] In the description of the present application, it should also be noted that, unless otherwise explicitly specified and limited, the terms "arrangement", "installation", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0029] Reference Figure 1 , 2, one embodiment of the present application is:

[0030] The short-term memory discrimination method based on multi-modal neuroimaging and deep learning comprises the following steps:

[0031] S1, synchronizing the EEG-fMRI device to collect the electroencephalogram signal and the BOLD signal in the resting state and the memory task;

[0032] S2, pre-processing the data collected in S1 step;

[0033] The data preprocessing specifically includes pre-processing of EEG signal and fMRI signal;

[0034] In the pre-processing of EEG signal, adaptive filtering and independent component analysis are used to remove MRI gradient artifact and ECG artifact;

[0035] In the pre-processing of fMRI signal, head motion correction, spatial standardization and smoothing processing are performed based on DPABI and SPM12.

[0036] S3, extracting feature data of EEG and fMRI through a convolutional neural network;

[0037] The feature data of the EEG includes time-frequency domain features of theta waves and time-frequency domain features of gamma wave power.

[0038] The feature data of the fMRI includes BOLD signals of the hippocampus and the prefrontal lobe.

[0039] The convolutional neural network is a depth separable convolutional neural network, and the separable convolutional neural network includes a depth convolution and a pointwise convolution.

[0040] The depth convolution first performs independent convolution processing on each channel of the input.

[0041] The pointwise convolution fuses the channels using a 1x1 convolution kernel.

[0042] S4, the feature data of the EEG and the fMRI are fused through position encoding and attention mechanism weighting to generate a joint feature vector.

[0043] Specifically, the attention mechanism incorporates the position encoding information into the attention, and uses the obtained position encoding information to quickly locate the position of interest. By focusing on the most important part of the image, the performance of the algorithm is improved without additional computational resources.

[0044] S5, classifying the joint feature vector based on an ELM algorithm to output a memory state discrimination result.

[0045] Since the output weights and biases of the ELM algorithm are randomly determined by an analytical method, it does not need to be iteratively trained like a traditional neural network, so the ELM algorithm is used to classify the joint feature vector, which has a faster training speed and better generalization ability than traditional neural networks.

[0046] As shown in FIG. Figure 2 In the experimental design, the subjects were randomly divided into two groups. All subjects first obtained resting-state EEG-fMRI data as a baseline, then performed pre-test content and post-test content, and the post-test content was a repetition of the pre-test content but used different vocabulary. The control group did not perform any operation between the two tests, and the experimental group learned the palace memory method between the two tests.

[0047] The order of the pre-test content and the post-test content is: baseline resting state, encoding phase, memory consolidation period resting state, recognition phase, data analysis.

[0048] Specifically, in the pre-test content, the baseline resting state: 8 minutes of resting state EEG-fMRI data is collected; the encoding stage: 12 groups of 72 words (6 words per group) are presented in the MRI, and the task state data is recorded; the memory consolidation period resting state: 8 minutes of resting state data is collected after encoding; the recognition stage: 12 pairs of words are presented, and the subject judges whether the order is consistent with the encoding stage, and the behavioral accuracy is recorded; data analysis: the association between the task state brain activity in the encoding stage and the memory performance is analyzed by generalized linear model (GLM), and the memory consolidation effect is predicted based on functional connection analysis (such as the strength of hippocampus-default network connection in resting state).

[0049] The post-test content repeats the above steps, but uses another 72 words.

[0050] More specifically, by constructing 4 paths, each path has 6 location pegs; in the encoding, first remember 3 rounds with one path, and then use the next path; in the recognition stage, the order of the words in each path is disturbed.

[0051] By designing a phased memory experiment task, fMRI and EEG data are collected and fused, and high-precision memory state classification is realized by combining CNN and ELM algorithms, and correlation analysis is performed with the behavioral results, and short-term memory characteristic signals are determined by neuroimaging data, and the neural mechanisms of memory encoding and consolidation are analyzed. It can be applied to cognitive evaluation, brain-computer interface and neuro-psychiatric disease diagnosis.

[0052] Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to part of the technical features, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A short-term memory discrimination method based on multi-modal neuroimaging and deep learning, characterized in that: The method comprises the following steps: S1, synchronizing an EEG-fMRI device to collect electroencephalogram signals and BOLD signals in resting state and memory tasks; S2, preprocessing the data collected in the step S1; S3, extracting feature data of the EEG and the fMRI through a convolutional neural network; S4, fusing the feature data of the EEG and the fMRI through position encoding and attention mechanism weighting to generate a joint feature vector; S5, classifying the joint feature vector based on an ELM algorithm to output a memory state discrimination result.

2. The short-term memory discrimination method based on multi-modal neuroimaging and deep learning according to claim 1, characterized in that: The data preprocessing comprises EEG signal preprocessing and fMRI signal preprocessing; The EEG signal preprocessing adopts adaptive filtering and independent component analysis to remove MRI gradient artifacts and electrocardiogram artifacts; The fMRI signal preprocessing is based on DPABI and SPM12 to perform head motion correction, spatial standardization and smoothing processing. 3.The short-term memory discrimination method based on multi-modal neuroimaging and deep learning of claim 1, wherein: In the step S3, the feature data of the EEG comprises time-frequency domain features of theta waves and time-frequency domain features of gamma wave power; The feature data of the fMRI comprises BOLD signals of the hippocampus and the prefrontal lobe. 4.The short-term memory discrimination method based on multi-modal neuroimaging and deep learning of claim 1, wherein: The convolutional neural network is a deep separable convolutional neural network, which comprises a depth convolution and a pointwise convolution; The depth convolution performs independent convolution processing on each channel of the input; The pointwise convolution fuses the channels through a 1x1 convolution kernel. 5.The short-term memory discrimination method based on multi-modal neuroimaging and deep learning according to claim 1, characterized in that: In the step S4, the attention mechanism integrates the position encoding information into the attention, and the obtained position encoding information is used to locate the interested position faster.

Images