A brain effect connection learning method based on spatiotemporal graph multi-head attention

Abstract

The application discloses a brain effect connection learning method based on a space-time graph multi-head attention, and the space-time graph multi-head attention network comprises a space graph multi-head attention block and a time multi-head attention block. First, data is embedded and then input into a first STGMHA network. In each intermediate STGMHA network, the output of the previous network is taken as the input of the next network, and the output of the last STGMHA network is input into a prediction layer. The application first uses GAT to extract a brain effect connection network from fMRI data. Unlike most technologies using GCN, the GCN uses a symmetric Laplacian matrix in the convolution process, and therefore cannot be directly applied to a directed graph. In contrast, the GAT can be applied to an undirected graph and more effectively extracts the causal relationship between brain regions.

Description

Technical Field

[0001] This invention relates to methods for extracting causal knowledge and spatiotemporal features from fMRI (functional magnetic resonance imaging) data. For the problem of brain disease diagnosis based on fMRI, a brain effect connectivity learning method based on spatiotemporal graph multi-head attention is designed. Background Technology

[0002] Brain effective connectivity networks (ECNs) reflect how one brain region influences another, and whether this influence is achieved through synaptic transmission or overall collective activity. ECNs can be viewed as a graph structure, with brain regions as nodes and edges representing the relationships of influence between different regions. Studying and analyzing ECNs allows for a better understanding of the functions and information transmission between brain regions, and their roles in various neural and cognitive processes. ECN analysis has been applied in numerous clinical research areas, providing in-depth research into neurological and psychiatric disorders, including Alzheimer's disease (AD), schizophrenia, depression, and autism spectrum disorder (ASD).

[0003] In recent years, many techniques have emerged for analyzing fMRI data, which can be mainly divided into two categories: model-based methods and data-driven methods. The difference between these two types of methods is that model-based methods require prior knowledge, while data-driven methods do not rely on specific theoretical models or assumptions.

[0004] Several common model-based methods exist, including structural equation modeling (SEM), dynamic causal modeling (DCM), and Granger causality (GC). It's important to note that GC methods require stationary data, necessitating transformation of non-stationary fMRI data to meet this requirement. Furthermore, GC methods cannot determine the directionality of these causal relationships. In contrast, DCM and SEM methods offer solutions for non-stationary fMRI data. DCM operates within a Bayesian framework, allowing researchers to infer causal relationships between brain regions and providing the ability to model dynamic changes in neural activity. On the other hand, SEM methods rely on multivariate linear equations, helping researchers build structural models to reveal relationships between brain regions and quantify the strength and directionality of these relationships through parameter estimation.

[0005] Model-based methods are popular in practice because they are easy to implement and interpret. However, data-driven methods tend to be more attractive when dealing with resting-state fMRI data where prior knowledge is lacking. In this regard, many model-based methods have been designed to estimate brain effector connectivity networks.

[0006] Data-driven approaches aim to discover patterns in brain effector connectivity networks directly from fMRI data without relying on prior knowledge. These methods utilize various techniques, such as independent component analysis (ICA), two-step methods, graph-based analysis, pwLiNGAM, CVAEEC, CR-VAE, DiffAN, and Bayesian network-based methods. These data-driven approaches promise to reveal the complex links between brain connectivity patterns and mental health without any prior knowledge. However, these data-driven approaches have some limitations. They often struggle to simultaneously and effectively capture both the broader global and complex local spatiotemporal properties of fMRI data.

[0007] In recent years, research and exploration into the relationship between effective brain connectivity and neurodegenerative diseases have increased significantly. Brain effector connectivity network analysis has been applied in multiple clinical research areas. By inferring insights into effector connectivity networks from neuroimaging data (such as fMRI), researchers can establish biomarkers for early detection, disease progression monitoring, and treatment efficacy assessment. This also allows for elucidating the interactions between different brain regions and their contributions to various cognitive processes and behaviors. Summary of the Invention

[0008] To address the limitation of current brain effect connectivity learning methods that fail to simultaneously consider both global and local spatiotemporal information in fMRI data, this paper proposes a novel brain-effective connectivity learning method, STGMHA, based on spatiotemporal graph multi-head attention networks. This method utilizes spatial multi-head attention and graph attention networks (GAT) to extract spatial features from fMRI data, while simultaneously employing temporal multi-head attention and temporal convolutional networks (TCN) to extract temporal features and long-term dependencies. After several rounds of extraction, these features are finally input into the prediction layer.

[0009] The main idea of ​​this invention is as follows: the spatiotemporal graph multi-head attention network includes a spatial graph multi-head attention block and a temporal multi-head attention block. First, the data is embedded and then input into the first STGMHA network. In each intermediate STGMHA network, the output of the previous network serves as the input of the next network, and the output of the last STGMHA network is input into the prediction layer.

[0010] A brain effect connectivity learning method based on spatiotemporal graph multi-head attention includes the following steps:

[0011] Step 1, Data Acquisition: To verify the effectiveness of the model proposed in this invention, experiments were conducted on the Sanchez dataset to evaluate the performance of the method.

[0012] Step 2, Data Preprocessing and Parameter Initialization: This method first preprocesses the data using a sliding window to perform data augmentation, and then uses the Pearson correlation coefficient to construct a brain effect connectivity network as the initial parameters of the brain effect connectivity network. At the same time, other hyperparameters are initialized.

[0013] Step 3, Input Embedding: This method maps the input fMRI data to a higher-dimensional space, changing the dimension of the input fMRI data from B×T×N to B×T×N×C.

[0014] Step 4, extract the brain effect connection network: input the data obtained from the input embedding into the causal strength and brain effect connection network of the spatial graph multi-head attention block enhancement nodes and related nodes.

[0015] Step 5, extracting temporal features: Long-term dependency features of fMRI time series data are extracted through temporal multi-head attention blocks.

[0016] Step six, repeat steps four and five: use the extracted features and brain effect connection network as input data, repeat steps four and five until the defined number of layers of the spatiotemporal graph multi-head attention network is reached.

[0017] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects;

[0018] (1) This invention is the first method to utilize GAT to extract brain effector connectivity networks from fMRI data. Unlike most techniques that use GCN, which employs a symmetric Laplacian matrix during convolution and therefore cannot be directly applied to directed graphs, GAT can be applied to undirected graphs and more effectively extracts causal relationships between brain regions.

[0019] (2) Research on brain connectivity networks using spatiotemporal graphs for multi-head attention learning. This method can capture not only spatial information between brain regions in fMRI data, but also temporal information with long-term dependencies.

[0020] (3) This method employs a novel spatial-temporal attention mechanism, utilizing spatial multi-head attention and GAT to extract spatial features and graph structures in different subspaces. In addition, this method also utilizes temporal multi-head attention to extract local temporal features in different subspaces and uses TCN to capture the global temporal context.

[0021] (4) Systematic experiments were conducted on simulated and real fMRI datasets, and it was found that this method can learn brain effect connectivity networks from fMRI datasets better than the state-of-the-art methods. Attached Figure Description

[0022] Figure 1 : Flowchart of the model training process involved in this method.

[0023] Figure 2 : The brain effect connectivity network learned by STGMHA on real resting-state fMRI data. Detailed Implementation

[0024] The specific implementation methods and detailed steps of the present invention are described below. The specific implementation process of the present invention is as follows: Figure 1 As shown, it specifically includes:

[0025] (Step 1) Data acquisition.

[0026] To verify the effectiveness of the proposed model, experiments will be conducted on the Sanchez dataset and a real resting-state fMRI dataset to evaluate its performance in extracting brain effect connectivity networks. The simulated dataset used is the Sanchez dataset, which has 5 nodes (brain regions), 60 samples (individual subjects), and 500 time steps. The real resting-state fMRI dataset used in this invention consists of 23 individual subjects, with each subject having 421 time steps. The data is bandpass filtered in the range of 0.008 to 0.08 Hz without spatial smoothing to avoid signal aliasing between adjacent regions. The following seven regions of interest (ROIs) are involved in the medial temporal lobe: the perispinal cortex divided into Brodmann Areas 36 (BA36) and 35 (BA35); the parahippocampal cortex (PHC); the entorhinal cortex (ERC); the subiculum (SUB); the cornua 1 (CA1); and the region consisting of the cornua 2, 3 and the dentate gyrus (CA23DG).

[0027] (Step 2) Data preprocessing and parameter initialization.

[0028] For STGMHA, a large dataset is required to ensure the reliability of the results. However, the size of fMRI datasets is limited, thus data augmentation of the fMRI data is necessary. In this method, a sliding window approach is used to implement data augmentation. Specifically, for a time series of length T, if the window size is set to W and the step size to S, then M time series data of length W can be obtained, where... Regarding parameter initialization, this method uses the Pearson correlation coefficient to construct a brain effect connectivity network as initial parameters, and simultaneously initializes other hyperparameters.

[0029] (Step 3) Input embedding.

[0030] This method uses 1×1 convolutional layer pairs to map the input fMRI data to a higher-dimensional space, changing the dimension of the input fMRI data from B×T×N to B×T×N×C.

[0031] (Step 4) Extract the brain effect connectivity network.

[0032] In the spatial graph multi-head attention block, spatial multi-head attention is first used to extract graph information of different dimensions from the fMRI data, and then GAT is used to extract the graph structure of the fMRI data. If two nodes have similar features in the graph, they have higher weight values ​​in the brain effect connectivity network. Therefore, after processing by the spatial graph multi-head attention network, a node will be enhanced by similar nodes. The resulting brain effect connectivity network can be used to predict the values ​​of other nodes.

[0033] Multi-head attention in space can simultaneously collect information from different subspaces, thereby enhancing the ability to capture different aspects of the input sequence. Multi-head attention consists of multiple self-attention mechanisms, each of which first obtains information through a linear transformation. :

[0034]

[0035] in This represents the input of the i-th spatial attention head after embedding the fMRI time-series data. Represents a linear transformation matrix. This represents a constant offset. Let represent the Query, Key, and Value of the i-th spatial attention head in the multi-head attention block of the spatial graph, respectively. In the calculation... Then, the attention weights are obtained using the following formula:

[0036]

[0037]

[0038] in This represents the dimension of the Query, Key, and Value vector. It is an activation function. This represents a neural network that illustrates the brain effects obtained from spatial multi-head attention. Finally, the output of the spatial graph multi-head attention block is obtained through a feedforward network and residual connections. :

[0039]

[0040] in, It is a feedforward neural network consisting of two linear layers and Activation function composition, This indicates residual join and normalization operations.

[0041] GAT (Graph Attention Network) is a deep learning model for processing graph data. It combines the ideas of attention mechanisms and graph neural networks to powerfully express the relationships between nodes. In GAT, the time-series data for each brain region is first linearly transformed, and then a weighted sum is performed using a self-attention mechanism to obtain the attention coefficients between each brain region.

[0042]

[0043] in This represents the attention weight function. This represents a linear transformation matrix. exist After processing by the activation function, through Activation functions reveal causal relationships between different brain regions:

[0044]

[0045] in yes The extension introduces a slope at negative values. This indicates the brain regions in the brain effect connectivity network extracted by GAT. brain regions The applied causal weighting coefficients. By obtaining causal relationships between brain regions, nodes... It can be represented by its adjacent brain region nodes, that is, those nodes that... Brain regions that exert causal influence. In the case of multi-head attention, these features need to be linked together:

[0046]

[0047] in Indicates series connection. It is a non-linear activation function. Represents a node Adjacent brain region nodes, This represents the linear transformation matrix. The data from each brain region node are then combined to obtain the fMRI data output by GAT:

[0048]

[0049] The output of the spatial graph multi-head attention block is obtained by weighted summing of the outputs of multi-head attention and GAT:

[0050]

[0051]

[0052] in It is a non-linear activation function. These represent the fMRI data from spatial multi-head attention and GAT outputs, respectively. Meanwhile, the weighting coefficients obtained from the above equation... The brain effect connectivity network of spatial multi-head attention block output is obtained by weighted summation of the brain effect connectivity networks of spatial multi-head attention and GAT output.

[0053] (Step 5) Extract time features.

[0054] In the temporal multi-head attention block, the first step is to extract local temporal information from multiple dimensions of the fMRI data using temporal multi-head attention. Subsequently, TCN is used to capture broader global temporal context information that appears in the fMRI data.

[0055] Since attention cannot understand the order and relative positions of the input sequence, position embedding (PE) is needed to self-include positional information in the input data.

[0056]

[0057] in This indicates the position of the vector in the fMRI time series. Indicates the position in the vector. This represents the dimension of a vector. The value is usually 10000.

[0058] Temporal multihead attention is similar to spatial multihead attention in spatial graph multihead attention blocks, but in fMRI time-series data... The dimensions have changed. ,therefore Self-attention can be calculated using the following formula:

[0059]

[0060] in This represents the dimension of the Query, Key, and Value vector. It is an activation function.

[0061] TCN is a convolutional network model based on a one-dimensional convolutional neural network, commonly used for processing time series data. A TCN consists of multiple residual connected blocks, each containing dilated causal convolution, weight normalization, ReLU activation, and dropout. For one-dimensional sequences... Convolutional filter with filter size K fMRI chronology data at time step t It can be obtained through the following formula:

[0062]

[0063] in It is the expansion factor. This represents a convolutional filter. By using dilated causal convolution, the receptive field of the network is expanded, and longer historical information from fMRI time-series data can be considered through a shallower network. Similar to the spatial graph multi-head attention block, the output of the temporal multi-head attention block is obtained by a weighted sum of the outputs of the multi-head attention and the TCN:

[0064]

[0065]

[0066] in It is a non-linear activation function. These represent fMRI data from temporal multi-head attention and TCN output, respectively.

[0067] (Step 6) Repeat steps 4 and 5.

[0068] The extracted fMRI time-series data and the brain effect connectivity network are used as input data. Steps four and five are repeated until the defined number of layers of the spatiotemporal graph multi-head attention network is reached.

[0069] To fully verify the superiority of this method, the proposed STGMHA algorithm was compared with existing methods such as pwLiNGAM, lsGC, two-Step, ACOCTE, and DiffAN on the Sanchez dataset. The results were evaluated using four metrics: Precision, Recall, Accuracy, and SHD. The algorithm was tested on data from 60 individual subjects in the Sanchez dataset, and the results are presented as mean ± standard deviation, as shown in Table 1.

[0070] Table 1 Comparison of methods on the Sanchez dataset

[0071]

[0072] According to the table above, pwLiNGAM, lsGC, and the two-Step method exhibit high recall but low precision, while ACOCTE and DiffAN show relatively low precision and recall. Conversely, STGMHA demonstrates the best performance in terms of precision, accuracy, and SHD. Notably, STGMHA significantly outperforms other methods in precision while exhibiting lower recall. This may be because STGMHA is a fraction-based method that introduces a sparsity penalty term for the brain effect connectivity network during fraction calculation, resulting in a relatively sparse brain EC network. In conclusion, STGMHA outperforms other algorithms, exhibiting the highest performance.

[0073] To compare the performance of these algorithms, experiments were conducted on a real resting-state fMRI dataset. Since there is no standard answer for brain effector connectivity networks in real fMRI data, existing work was used as a reference for model evaluation. Experiments were performed on the left and right hemispheres of all 23 subjects using both the STGMHA and baseline methods, and the brain effector connectivity networks obtained for each subject were averaged to obtain the final brain effector connectivity network. Figure 2 The brain effect connectivity network obtained from 7 ROIs is presented. The figure shows that STGMHA found 12 correct connections, 4 spurious connections, and lost 5 correct connections in the left hemisphere. In the right hemisphere, it found 13 correct connections, 3 spurious connections, and lost 4 correct connections. Overall, STGMHA found the most correct connections and missed the fewest correct connections in the right hemisphere, ranking second in other metrics. Regarding the baseline method, although it ranked first in one evaluation metric, it performed poorly in other metrics.

[0074] The above experiments show that STGMHA has significant advantages over other methods, and therefore has great application prospects in the diagnosis of brain diseases.

Claims

1. A brain effect connectivity learning method based on spatiotemporal graph multi-head attention, characterized in that, Includes the following steps: Step 1, Data Acquisition: Conduct experiments on the Sanchez dataset; Step 2, Data Preprocessing and Parameter Initialization: This method first uses a sliding window to preprocess the data and perform data augmentation. Then, the Pearson correlation coefficient is used to construct the brain effect connectivity network as the initial parameter of the brain effect connectivity network. At the same time, other hyperparameters are initialized. Step 3, Input Embedding: Map the input fMRI data to a higher-dimensional space, so that the dimension of the input fMRI data changes from B×T×N to B×T×N×C; Step 4, extract the brain effect connectivity network: input the data obtained from the input embedding into the causal strength and brain effect connectivity network of the spatial graph multi-head attention block enhancement nodes and related nodes; Step 5, Extracting temporal features: Long-term dependency features of fMRI time series data are extracted through temporal multi-head attention blocks; Step six, repeat steps four and five: use the extracted features and brain effect connection network as input data, repeat steps four and five until the defined number of layers of the spatiotemporal graph multi-head attention network is reached; In step four, the spatial graph multi-head attention block first uses spatial multi-head attention to extract graph information of different dimensions of fMRI data, and then uses GAT to extract the graph structure of fMRI data; Multi-head attention consists of multiple self-attention mechanisms. In each self-attention mechanism, a linear transformation is first performed to obtain... : ； in This represents the input of the i-th spatial attention head after embedding the fMRI time-series data. Represents a linear transformation matrix. Indicates a constant offset; These represent the Query, Key, and Value of the i-th spatial attention head in a multi-head attention block of a spatial graph; in the computation Then, the attention weights are obtained using the following formula: ； ； in This represents the dimension of the Query, Key, and Value vector. It is an activation function. The network represents the brain effect connection obtained by spatial multi-head attention; finally, the output of the spatial graph multi-head attention block is obtained through a feedforward network and residual connections. : ； in, It is a feedforward neural network consisting of two linear layers and Activation function composition, This represents residual join and normalization operations; In GAT, the time-series data for each brain region are first linearly transformed, and then the data are weighted and summed using a self-attention mechanism to obtain the attention coefficient between each brain region. ； in, This represents the attention weight function. Represents a linear transformation matrix; exist After processing by the activation function, through Activation functions reveal causal relationships between different brain regions: ； in yes The extension introduces a slope at negative values. This indicates the brain regions in the brain effect connectivity network extracted by GAT. brain regions The applied causal weighting coefficients; by obtaining causal relationships between brain regions, nodes Represented by its adjacent brain region nodes, i.e., those that... Brain regions that exert causal influence; in the case of multi-head attention, these features are linked together: ； in Indicates series connection. It is a non-linear activation function. Represents a node Adjacent brain region nodes, This represents the linear transformation matrix; the data from each brain region node are combined to obtain the fMRI data output by GAT: ； The output of the spatial graph multi-head attention block is obtained by weighted summation of the outputs of multi-head attention and GAT. ； ； in It is a non-linear activation function. These represent the fMRI data from spatial multi-head attention and GAT outputs, respectively; meanwhile, the weighting coefficients obtained from the above formula... The brain effect connectivity network of spatial multi-head attention block output is obtained by weighted summation of the brain effect connectivity networks of spatial multi-head attention and GAT output.

2. The brain effect connectivity learning method based on spatiotemporal graph multi-head attention according to claim 1, characterized in that, In step two, a sliding window method is used to perform data augmentation. For a time series of length T, if the window size is set to W and the step size to S, then M time series data of length W are obtained, among which... A brain effect connectivity network was constructed using Pearson correlation coefficients as initial parameters.

3. The brain effect connectivity learning method based on spatiotemporal graph multi-head attention according to claim 1, characterized in that, In step five, within the temporal multi-head attention block, local temporal information is extracted from multiple dimensions of the fMRI data using temporal multi-head attention; subsequently, TCN is used to capture broader global temporal context information appearing in the fMRI data. Location embedding (PE) enables location information to be automatically included in the input data: ； in This indicates the position of the vector in the fMRI time series. Indicates the position in the vector. This represents the dimension of a vector. The value is 10000; Temporal multihead attention is similar to spatial multihead attention in spatial graph multihead attention blocks, but in fMRI time-series data... The dimensions have changed. ,therefore Self-attention is calculated using the following formula: ； in This represents the dimension of the Query, Key, and Value vector. It is an activation function; TCN consists of multiple residual connect blocks, each containing dilated causal convolution, weight normalization, ReLU activation function, and dropout; for a one-dimensional sequence Convolutional filter with filter size K fMRI chronology data at time step t Obtained through the following formula: ； in It is the expansion factor. This represents a convolutional filter; the output of the temporal multi-head attention block is obtained by a weighted sum of the outputs of the multi-head attention and the TCN: ； ； in It is a non-linear activation function. These represent fMRI data from temporal multi-head attention and TCN output, respectively.

Images