A Deep Learning Method for Dynamic Brain Effective Connectivity by Integrating fMRI and EEG Data

Abstract

The present invention discloses a deep learning method for dynamic brain effective connectivity integrating fMRI and EEG data. First, the method constructs a dynamic causal network to effectively extract causal information from the two modalities of fMRI and EEG. An attention mechanism is introduced, which guides the update of causal information of the two modalities by generating multi-modal attention maps, thereby obtaining accurate DEC. The method also designs three evaluation networks, namely an fMRI evaluation network, an EEG evaluation network, and a modality evaluation network, to calculate their respective gradients respectively, and further guides the dynamic causal network to obtain more accurate DEC through weighted combination. The present invention designs an fMRI evaluation network, an EEG evaluation network, and a modality evaluation network, which improves the accuracy and performance of algorithm learning.

Description

Technical Field

[0001] The present invention relates to a method for causal discovery of fMRI functional magnetic resonance imaging data and EEG electroencephalogram data. Aiming at the goal of realizing computer-aided diagnosis of brain diseases based on fMRI and EEG data, a dynamic brain effect connectivity deep learning method integrating fMRI and EEG data is designed. Background Art

[0002] A brain effective connectivity network can reflect the potential causal relationships between different brain regions of the human brain and is an effective tool for early diagnosis and treatment evaluation of brain diseases. In recent years, deep learning-based brain effect connectivity learning methods have shown great potential. Since Dynamic Effective Connectivity (DEC) can characterize the temporal changes of the effect connectivity between brain regions and contains richer dynamic brain nerve activity information. Therefore, DEC learning is a research hotspot in the current field of brain science.

[0003] In recent years, with the popularization of neuroimaging technologies, more and more detection imaging technologies have been applied to the exploration of human brain functions. Among them, the most widely used are functional Magnetic Resonance Imaging (fMRI) and Electroencephalogram (EEG). The advantages of fMRI and EEG are that they can not only non-invasively detect the dynamic changes of brain functions, but also there are no known side effects at present. Therefore, more and more brain science researchers use fMRI data and EEG data to study human brain functions and have revealed a series of associations between brain nerve activity laws and human behaviors.

[0004] The main defect of the existing DEC learning methods is that only single fMRI or EEG data is used for learning. Among them, fMRI data has the characteristics of high spatial resolution and low temporal resolution, while the characteristics of EEG data are the opposite. Since both the temporal and spatial information of brain nerve activities will affect the learning performance of DEC, one-sided single-modal data makes it difficult for the existing methods to accurately learn DEC.

[0005] Therefore, how to accurately learn a dynamic brain effect connectivity network from multiple-modal data of fMRI and EEG is still a very challenging topic in this field. Summary of the Invention

[0006] In view of the situation that the accuracy of the brain effect connection network learned from single-modal data by existing dynamic brain effect connection learning methods is not high enough, the present invention proposes a deep learning method for dynamic brain effect connection that fuses fMRI and EEG data. This method first effectively extracts causal information in both fMRI and EEG modalities by constructing a dynamic causal network. Then, this method introduces an attention mechanism, which guides the update of causal information in both modalities by generating multi-modal attention maps, and further obtains accurate DEC. In addition, this method also designs three evaluation networks, namely the fMRI evaluation network, the EEG evaluation network, and the modality evaluation network, to calculate their respective gradients respectively, and further guides the dynamic causal network to obtain more accurate DEC through weighted combination.

[0007] The main idea of implementing the present invention is as follows: fMRI data has high spatial resolution but low temporal resolution. On the contrary, EEG data has the characteristic of high temporal resolution, while its spatial resolution is at a lower level compared with fMRI data. For the DEC learning task, both the temporal and spatial information of brain nerve activities is of great significance. Using only single-modal data causes these methods to be unable to comprehensively utilize rich temporal and spatial information, thus limiting the further improvement of their learning performance. Therefore, using data from both modalities simultaneously can integrate the temporal and spatial information of brain nerve activities. By training a deep learning neural network to update parameters, it can stimulate the neural network to model the interaction patterns of multi-modal information, enabling the network to adaptively fuse multi-modal data. Therefore, applying deep learning in the present invention to learn DEC from fMRI and EEG data in both modalities may help alleviate the problem that single-modal data is difficult to provide sufficient information, thereby further improving the accuracy of dynamic brain effect connection learning.

[0008] A deep learning method for dynamic brain effect connection that fuses fMRI and EEG data includes the following steps:

[0009] Step 1, obtain the source-level time series: Map the input fMRI and EEG time series data into source-level time series with the same dimension using the fMRI and EEG source coding modules.

[0010] Step 2, extract causal information: Use a dynamic causal network (DCN) to extract causal information from the source-level time series of both fMRI and EEG modalities, and generate the generated time series of both modalities.

[0011] Step 3, Time series fusion: An Attention-guided Layer is used to generate multi-modal attention maps of two modalities and fuse time series, and they are respectively fused with the causal relationship estimated by DCN and the generated time series to obtain DECs of two modalities and posterior source level time series.

[0012] Step 4, Time series decoding and state partitioning: Use the fMRI and EEG signal decoding modules to restore the posterior source level time series to the posterior time series with the same dimension as the input data, and use the state estimation module to partition the fused DECs of two modalities.

[0013] Step 5, Calculate gradients and output DEC: First, use three evaluation networks, namely the fMRI evaluation network, the EEG evaluation network, and the modality evaluation network, to calculate their respective gradients, and finally obtain the final DEC through a weighted combination guidance algorithm for the output results of the three evaluation networks.

[0014] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects;

[0015] (1) Introduce a multi-modal fusion mechanism, utilize data of both fMRI and EEG modalities simultaneously, integrate the temporal and spatial information of cranial nerve activities, and alleviate the problem that it is difficult for single-modal data to provide sufficient information.

[0016] (2) Design an attention guidance mechanism, which plays a certain filtering role for redundant information of two modalities, and effectively fuses data of two modalities to learn DEC.

[0017] (3) Design the fMRI evaluation network, the EEG evaluation network, and the modality evaluation network, which improve the accuracy and performance of algorithm learning. Brief description of the drawings

[0018] Figure 1 : Flow chart of this method.

[0019] Figure 2 : DEC obtained by learning on the real dataset in the car and house task states by this method. Detailed implementation manners

[0020] The following elaborates on the detailed implementation manners and steps of the present invention. The specific implementation process of the present invention is as Figure 1 shown, specifically including:

[0021] (Step 1) Obtain the source level time series.

[0022] The present invention designs an fMRI source coding module and an EEG source coding module to map the data of the two modalities into the same feature space. These two source coding modules provide solutions to the inverse problems of the two modalities. The fMRI time series x f and the EEG time series x e are respectively input into the source coding modules, and the fMRI source-level time series r f and the EEG source-level time series r e are respectively output.

[0023] The BOLD inverse problem of fMRI source coding is defined as:

[0024] r f = x f L + ε f (1)

[0025] where is the fMRI time series measured from P f brain regions over a time length of T f . is the fMRI source-level time series, and T e is the duration of the EEG time series. is the BOLD response matrix composed of trainable neural network parameters, and ε f is Gaussian noise.

[0026] The purpose of the inverse problem of EEG source coding is to estimate the source location of the measured neural signals, which is defined as:

[0027] r e = Jx e + ε e (2)

[0028] where is the EEG time series measured from P e electrodes over a time length of T e . is the EEG source-level time series. is the lead field matrix composed of trainable neural network parameters, and ε e is Gaussian noise.

[0029] According to the source coding modules of the two modalities, the fMRI source-level time series r f and the EEG source-level time series r e can be obtained. Concatenating r f and r e can yield the hybrid representation of the two modalities Step 2: Causal information extraction.

[0030] First, the data is discretized. The present invention designs a dynamic causal network to extract the time-varying causal relationships between brain regions in two modalities, and the source-level time series r f or r e generated in Step 1 is input into the DCN, and the generated time series is output or The framework of the DCN is similar to a set of recurrent neural networks, and it contains P f ×T e causal units (CausalUnit, CU). The number of rows of the hidden layer matrix formed by the CUs is equal to the number of brain regions P f , and the number of columns is the same as the length T e of the time series, and the network parameters between the hidden layers do not share weights.

[0031] The forward propagation process of the CU is expressed as:

[0032]

[0033] The trainable neurons W = {W(1), W(2)…, W(T e )} of the DCN are the causal relationships between brain regions during T e . Therefore, the causal relationships W f and W e between all brain regions can be obtained using the DCNs of the two modalities.

[0034] (Step 3) Time series fusion.

[0035] The present invention introduces an attention guidance layer to facilitate the effective fusion of information from two modalities and strengthen the cross-information in the two modalities. These two attention guidance layers have the same network structure but do not share parameters. The inputs to the attention guidance layer are the query vector Q, the key vector K, and the value vector V. Among them, the query vector Q is obtained by transforming the mixed vector r mm from the two modalities through a single-layer fully connected layer, and W and V are obtained by transforming r f and r e respectively. This attention guidance layer can perform the interaction between the two modalities while fully retaining the information of their respective modalities, and the generated multi-modal attention map can effectively guide the update of the causal information of the two modalities.

[0036] First, the multi-modal attention map Att = {Att(1), Att(2)…, Att(T e )} can be obtained through the query vector Q and the key vector K, and the calculation formula is as follows:

[0037]

[0038] Among them, is the scaling factor, and its size is set to the dimension of the keyword vector K.

[0039] Then, multiply the attention Att by V to obtain the fused time series of fMRI and the fused time series of EEG The calculation formula is as follows:

[0040]

[0041] Finally, according to the fused time series obtained from the attention guidance layer and the generated time series obtained from the DCN Add these two vectors to obtain the posterior source level time series

[0042] (Step 4) Time series decoding and state partitioning

[0043] First, input the posterior source level time series and into the fMRI signal decoding module and the EEG signal decoding module. Then, obtain G by summing Att and W, aiming to use the multimodal attention map Att to guide the causal relationship W so as to narrow the gap between the two modalities. In addition, calculate the average value of G f and G e as the effective connectivity network G estimated by MCAN mm ={G mm (1), G mm (2)···, G mm (T e )}.

[0044] The present invention designs an fMRI signal decoding module and an EEG signal decoding module, and this decoding process is the inverse process of source coding. Among them, the decoding of the fMRI source level time series is defined as:

[0045]

[0046] where is the fMRI source level time series of P e brain regions with time length T. f is the posterior fMRI signal. is the fMRI decoding matrix composed of trainable neural network parameters, and ε f is Gaussian noise.

[0047] The decoding of the EEG source level time series is defined as:

[0048] ​

[0049] Among them is the time length P e of the EEG source level time series of P electrodes. f is the posterior EEG signal. is the EEG decoding matrix composed of trainable neural network parameters, and ε is Gaussian noise. e is Gaussian noise.

[0050] During the state estimation process, first, the DEC network obtained from the above process is compressed into brain state features. Specifically, the DEC network G at each moment mm from P f ×P f is compressed into a 1×P f 2 dynamic brain state feature g. Then, the Pruned Exact Linear Time (Pelt) algorithm with an L2 cost function and an L1 penalty function is used to estimate the transition moments of the brain state. Specifically, the dynamic brain state feature is used as the partitioning object, and the set of transition moments cp(t * ) at each moment t * is found recursively until all moments are traversed and cp(T e ) is obtained, which is the final optimal set of transition moments Γ.

[0051] (Step six) Calculate the gradient and output DEC

[0052] The multi-modal causal evaluation network performs a joint classification task by three evaluation networks, including the fMRI evaluation network, the EEG evaluation network, and the modality evaluation network. How the three evaluation networks calculate the loss will be introduced below.

[0053] The fMRI discriminative network and the EEG discriminative network output the authenticity probability to discriminate whether the sample is the original time series or the posterior time series. The loss functions of the fMRI and EEG discriminators are respectively defined as:

[0054]

[0055]

[0056] Among them, represents the fMRI discriminator, represents the EEG discriminator. In addition, the sparsity penalties Φ f and Φ e are L2 regularization terms.

[0057] The modality discrimination network is used to distinguish whether the sample is an fMRI posterior time series or an EEG posterior time series, and the loss function is defined as:

[0058]

[0059] where is defined as:

[0060]

[0061] where represents matrix multiplication. The loss function of this method is calculated by the sum of three evaluation networks:

[0062]

[0063] where α, β, and γ are weight coefficients and belong to the hyperparameters of this method. Finally, the weighted combination of the output results of the three evaluation networks is used to output the final DEC.

[0064] To fully verify the superiority of this method, this method was compared with three existing DEC learning methods, namely swGC, nsDBN, and nPSE-GAN, on two generated fMRI and EEG simulation datasets with different brain connection states and real datasets. The real synchronous fMRI-EEG data can be freely downloaded at https: / / www.dropbox.com / sh / 15sltkkwb08w7xg / AADYY6PqzKeegxWu-AWbnfGXa?dl=0. The visual classification task dataset of car and house selection was used, which contains the data of 10 subjects. The subjects need to measure fMRI and EEG data simultaneously when performing the task. Among them, the number of EEG channels is 34, and the repetition time of fMRI is 2s. The effective connectivity network of face and house is the average value of the learning results of this method for 40 repeated experiments of 10 subjects.

[0065] For the experiments conducted on the simulation datasets, three evaluation metrics, namely Precision, Recall, and SHD, were used to evaluate the results. These three evaluation metrics are widely used in the evaluation of brain network learning. The experimental results are shown in Table 1.

[0066] Table 1: Performance comparison of MCAN and three DEC learning methods on simulation datasets

[0067]

[0068]

[0069] As can be seen from Table 1, the Recall of this method on all simulated datasets is higher than that of the other three comparison methods. This fully demonstrates that this method can accurately learn DEC. The performance of SHD and Precision on some datasets is slightly lower than that of other comparison algorithms, indicating that this method has learned some extra edges. In summary, compared with other DEC learning methods, this method has achieved excellent results on simulated data.

[0070] Next, a comparative experiment was conducted on the real dataset of the visual classification task of car and house selection. Since there is no standard network for the real dataset, the effect connection results of this method and three comparison methods were compared, as Figure 2 shown. In Figure 2 , gray indicates the existence of an effect connection from the region of interest in the row to the region of interest in the column. The number of brain effect connections in the car task state is significantly less than that in the house task state. This finding is consistent with some existing research results. In addition, in the house task state, the number of connections identified by this method is less, which may be because the attention guidance mechanism of this method filters the redundant information of the two modalities. When the subject views the image of the house, the connectivity of FFA and PPA increases respectively. This is because the FFA region processes information related to facial stimuli, and the brain nerve activity in the PPA region is related to the recognition of houses or locations. Thus, it can be seen that the DEC network estimated by this method can explain the human brain nerve activity.

[0071] The above experiments show that this method can play a certain filtering role in the redundant information of the two modalities, can effectively fuse the data of the two modalities to learn DEC, and its accuracy is better than that of existing other methods. Therefore, it has great application prospects in computer-aided brain disease diagnosis.

Claims

1. A dynamic brain effective connectivity deep learning method that fuses fMRI and EEG data, characterized in that, It includes the following steps: Step 1, obtaining the source-level time series: Using the fMRI source coding module and the EEG source coding module, map the input fMRI and EEG time series data into source-level time series of the same dimension for the two modalities; Step 2, extracting causal information: Utilize the Dynamic Causal Network (DCN) to extract causal information from the source-level time series of the two modalities of the fMRI source coding module and the EEG source coding module, and generate the generated time series of the two modalities; Step 3, time series fusion: Adopt an attention-guided layer to generate multi-modal attention maps and fused time series for the two modalities, and fuse them with the causal relationship estimated by the DCN and the generated time series respectively to obtain the DEC and posterior source-level time series of the two modalities; Step 4, time series decoding and state partitioning: Use the fMRI and EEG signal decoding modules to restore the posterior source-level time series to the posterior time series with the same dimension as the input data, and adopt a state estimation module to perform state partitioning on the fused DEC of the two modalities; Step 5, calculating the gradient and outputting the DEC: First, use three evaluation networks, namely the fMRI evaluation network, the EEG evaluation network, and the modality evaluation network, to calculate their respective gradients, and finally obtain the final DEC through a weighted combination guidance algorithm for the output results of the three evaluation networks; 2. A dynamic brain effective connectivity deep learning method integrating fMRI and EEG data according to claim 1, characterized in that In the fMRI source coding module and the EEG source coding module, the fMRI time series x f and the EEG time series x e are respectively input into the source coding module, and the fMRI source level time series r f and the EEG source level time series r e ; The BOLD inverse problem of fMRI source coding is defined as: r f = x f L + ε f (1) Among them is the fMRI time series measured for P f brain regions over a time length of T f ; is the fMRI source-level time series, and T is the fMRI source-level time series, and T e is the duration of the EEG time series; is the BOLD response matrix composed of trainable neural network parameters, and ε f is Gaussian noise The purpose of the inverse problem of EEG source coding is to estimate the source location of the measured neural signals, and it is defined as: r e = Jx e + ε e (2) where is the EEG time series measured by P e electrodes over a time length of T e ; is the EEG source level time series; is the lead field matrix consisting of trainable neural network parameters, and ε e is Gaussian noise; e Obtain the fMRI source-level time series r according to the source coding modules of two modalities f and the EEG source-level time series r e ; Concatenating r f and r e can obtain the hybrid representation of the two modalities 3. A dynamic brain effective connectivity deep learning method for integrating fMRI and EEG data according to claim 2, characterized in that In Step 2, the data is discretized, and a dynamic causal network is designed to extract the time-varying causal relationships between brain regions in two modalities. The source-level time series r f or r e generated in Step 1 is input into the DCN, and the generated time series is output or The framework of the DCN is similar to a set of recurrent neural networks and contains P f ×T e causal units CU; the number of rows of the hidden layer matrix formed by the CUs is equal to the number of brain regions P f , and the number of columns is the same as the length T e of the time series. The network parameters between the hidden layers do not share weights; The trainable neurons W of DCN = {W(1), W(2)…, W(T e )} are the causal relationships between brain regions during T e ; The causal relationships W between all brain regions are obtained using the DCNs of two modalities f and W e .

4. A dynamic brain effective connectivity deep learning method for fusing fMRI and EEG data according to claim 3, characterized in that, In step 3, an attention guidance layer is introduced to fuse the information of the two modalities and strengthen the cross-information in the two modalities; the two attention guidance layers have the same network structure; the inputs of the attention guidance layer are the query vector Q, the keyword vector K, and the value vector V; among them, the query vector Q is the mixed vector r of the two modalities mm obtained through a single-layer fully connected layer transformation, and K and V are respectively obtained through the transformations r f and r e obtained; First, a multi-modal attention map Att = {Att(1), Att(2), …, Att(T e )} is obtained by query vector Q and keyword vector K, and the calculation formula is as follows: Among them, is a scaling factor, and its size is set to the dimension of the keyword vector K; Then, multiply the attention Att by V to obtain the fused time series of fMRI and the fused time series of EEG The calculation formula is as follows: T = VAtt T (5) Finally, based on the fused time series obtained from the attention guidance layer and the generated time series obtained from the DCN add these two vectors to obtain the posterior source level time series 5. A dynamic brain effective connectivity deep learning method for fusing fMRI and EEG data according to claim 4, characterized in that In step 4, the posterior source level time series and are input into the fMRI signal decoding module and the EEG signal decoding module; then G is obtained by summing Att and W, aiming to use the multimodal attention map Att to guide the causal relationship W so as to narrow the gap between the two modalities; in addition, the average value of G f and G e is used as the effective connectivity network estimated by MCAN G mm = {G mm (1), G mm (2) ···, G mm (T e )}; Design the fMRI signal decoding module and the EEG signal decoding module. The decoding process is the inverse process of source coding; among them, the decoding of the fMRI source-level time series is defined as: Among them is T e the fMRI source level time series of P f brain regions; is the posterior fMRI signal; is the fMRI decoding matrix composed of trainable neural network parameters, ε f is Gaussian noise; The decoding of the EEG source-level time series is defined as: Among them is T e the EEG source level time series of the time length P f for P electrodes; is the posterior EEG signal; is the EEG decoding matrix composed of trainable neural network parameters, ε e is Gaussian noise; During the state estimation process, the obtained DEC network is compressed into brain state features: the DEC network G at each moment mm from P f ×P f is compressed into a 1×P f 2 dynamic brain state feature g; Use a pruning exact linear time algorithm with an L2 cost function and an L1 penalty function to estimate the transition moments of the brain state: Take the dynamic brain state features as the partitioning objects, and recursively find the set of transition moments cp(t * ) at each moment t * until all moments are traversed and cp(T e ) is obtained, which is the final optimal set of transition moments Γ.

6. A dynamic brain effective connectivity deep learning method for fusing fMRI and EEG data according to claim 1, characterized in that In Step 5, the multi-modal causal evaluation network performs a joint classification task by three evaluation networks, including the fMRI evaluation network, the EEG evaluation network, and the modality evaluation network; The fMRI discriminant network and the EEG discriminant network output the authenticity probability to discriminate whether the sample is the original time series or the posterior time series; the loss functions of the fMRI and EEG discriminators are respectively defined as: Among them, represents the fMRI discriminator, represents the EEG discriminator; the sparsity penalty Φ f and Φ e are L2 regularization terms; The modality discriminant network is used to distinguish whether the sample is the fMRI posterior time series or the EEG posterior time series, and the loss function is defined as: wherein is defined as: Among them represents matrix multiplication; the loss function of this method is calculated by the sum of three evaluation networks: Among them, α, β, and γ are weight coefficients, and the final DEC is output through a weighted combination of the output results of the three evaluation networks.

Images