A multimodal brain network important region identification method based on self-supervised Graph-Transformer

Abstract

The present invention discloses a method for identifying important areas of a multimodal brain network based on a self-supervised Graph‑Transformer. First, the MRI data of the brain is constructed into a multimodal brain map, and the structural information and functional information of the brain are mapped to the edge features and point features of the graph respectively; then, important brain map nodes are identified through a self-supervised model (GR‑GT). GR‑GT adopts an encoder-decoder architecture, takes the brain map as input and outputs a reconstructed brain map, requires the reconstructed brain map to be as close to the original brain map as possible, and extracts the contribution scores of all brain map nodes to the reconstruction task. Nodes with high contribution scores are more important to the reconstruction task and are the important brain areas identified by the present invention. The present invention analyzes data in a completely data-driven manner rather than by manual features, thereby identifying important areas of the brain network, effectively avoiding the inherent limitations brought by prior knowledge.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition, and in particular relates to a method for identifying important regions of a multimodal brain network. Background Art

[0002] The human brain is an extremely complex system that coordinates human cognition, behavior, and more. Studying key regions within brain networks is of great significance in the field of brain imaging, helping to better understand the brain's workings. The brain involves numerous connections between neurons, and non-invasive brain imaging techniques such as MRI facilitate brain modeling and analysis. Long-standing research has shown that a set of key regions within brain networks plays a crucial role. Existing work generally relies on prior knowledge from graph theory, such as exploring regions with high centrality or forming a rich club as key brain regions. However, these studies explore key brain regions based on prior knowledge rather than the brain data itself. Given that the true architecture of brain networks is not yet fully understood, over-reliance on prior knowledge may overlook the intrinsic characteristics of the brain networks themselves, resulting in inadequate identification of key brain regions. Summary of the Invention

[0003] In order to overcome the deficiencies of the prior art, the present invention provides a method for identifying important areas of a multimodal brain network based on a self-supervised Graph-Transformer. First, the MRI data of the brain is constructed into a multimodal brain map, and the structural information and functional information of the brain are mapped to edge features and point features of the graph respectively; then, important brain map nodes are identified through a self-supervised model (GR-GT). GR-GT adopts an encoder-decoder architecture, takes the brain map as input and outputs a reconstructed brain map, requires the reconstructed brain map to be as close to the original brain map as possible, and extracts the contribution scores of all brain map nodes to the reconstruction task. Nodes with high contribution scores are more important to the reconstruction task and are the important brain areas identified by the present invention. The present invention analyzes data in a completely data-driven manner rather than by manual features, thereby identifying important areas of the brain network, effectively avoiding the inherent limitations brought about by prior knowledge.

[0004] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0005] Step 1: Construct a multimodal brain map from brain MRI data;

[0006] The brain is defined as an undirected graph G = {V, E}, V = {v i |i∈1,2,…,N} represents the node set of the graph, which has N nodes; Represents the node features of the graph; Represents the adjacency matrix of the graph;

[0007] First, the brain surface is divided into N regions using a brain partition template, which serve as nodes of the graph. Then, using functional MRI signals, the Pearson correlation coefficient of the functional signals between nodes is calculated to obtain a functional similarity matrix, which is used as the node feature matrix X of the graph. The length of the eigenvector of each node is D = N. Finally, using diffusion tensor imaging (DTI) data, the fiber connections between nodes are calculated to obtain a structural connectivity matrix, which is then sparsified and binarized to obtain the graph adjacency matrix E.

[0008] Step 2: Obtain the contribution score of the brain node through the self-supervised model GR-GT module;

[0009] In the self-supervised model GR-GT module, an encoder-decoder structure is used to implement the graph reconstruction task and obtain the contribution score of the node;

[0010] First, the brain map obtained in step 1 is used as input, and the encoder module is used to extract the node representation of the graph. Then, the mask score module is used to obtain the contribution score of the node, and finally the decoder module is used to reconstruct the node features of the graph.

[0011] Encoder module: Combine Transformer with graph convolutional neural network (GCN). First, the Transformer layer is used to apply the self-attention mechanism to all nodes of the graph. Then, GCN is used to combine the topological information of the graph. The process is shown in Equations (1) and (2):

[0012] H′ (k+1) =Attention V (H (k) Q (k) , H (k) K (k) , H (k) V (k) ) (1)

[0013] H (k+1) =GCN(E,H′ (k+1) ) (2)

[0014] Among them H (k) is the graph node representation of the kth layer in the encoder module, V is the set of all nodes in the graph, is the projection matrix, Attention V Indicates the use of self-attention mechanism for all nodes of the graph, and E represents the adjacency matrix of the graph;

[0015] Mask score module: This module is placed after the encoder module and before the decoder module; the output of the encoder module is the node representation of the graph As the input of the mask score module; after passing through the multi-layer perceptron MLP and softmax, Z is mapped to p is regarded as the contribution score of the graph node; then, according to the ratio k, the nodes with lower scores are masked and assigned a mask representation vector; finally, the Hadamard product operation is performed on the node representation of the graph and the contribution score, which is used as the input of the decoding module; the calculation process is shown in formula (3):

[0016] p = softmax(MLP(Z))

[0017] idx = rank(p, 1-k)

[0018]

[0019]

[0020] Where softmax is a nonlinear function, rank represents the ranking function, idx represents the index with the highest value after sorting p, q represents the mask representation vector of the masked node, cat represents the concatenation of the node representation and the mask representation vector in the original order, ⊙ represents the Hadamard product, and Z′ represents the input of the decoder module;

[0021] Decoder module: The input of the decoder module is the graph node representation with mask information, and the output is the node features of the reconstructed graph; the decoder module has the same structure as the encoder module;

[0022] The GR-GT module reconstructs the graph by predicting the node features of the graph; the loss function is the mean square error (MSE) between the feature matrix Y of the reconstructed graph nodes and the feature matrix X of the input graph nodes;

[0023] Step 3: The dataset is passed through steps 1 and 2 to obtain the contribution scores of the reconstructed graph and nodes. The brain regions are ranked according to the scores, and the regions with the highest scores are selected as important brain regions according to the set ratio.

[0024] Preferably, the detailed expression of formula (4) is:

[0025]

[0026] in is the representation vector of node v, Trans and Att represent MLP and multi-head attention respectively, and Lnorm represents normalization;

[0027] The detailed expression of Att is:

[0028]

[0029] in is the attention weight between nodes v and v′.

[0030] Preferably, the detailed expression of formula (5) is:

[0031]

[0032] Where I is the identity matrix, σ is the nonlinear function, and W is the weight matrix.

[0033] Preferably, during training, the GR-GT module sets the number of layers in the encoder module and the decoder module to 1, the number of heads in the multi-head attention to 4, the number of epochs to 200, and uses the Adam optimizer to update the parameters, with the node mask ratio k being 0.5.

[0034] The beneficial effects of the present invention are as follows:

[0035] 1. This invention is of great significance for the identification of important areas of brain networks. It is a self-supervised deep learning model based on graph reconstruction. It analyzes data in a completely data-driven manner rather than using manual features to identify important areas of brain networks, effectively avoiding the inherent limitations brought by prior knowledge.

[0036] 2. Key regions of multimodal brain networks play a crucial role in understanding brain mechanisms and analyzing brain diseases. Analyzing the functional and structural characteristics of the brain helps us understand the brain's functions of information transmission, separation, and integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of the SSGR-GT framework flow of the present invention.

[0038] Figure 2 This is a schematic diagram of the GR-GT model network structure proposed in this invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below with reference to the accompanying drawings and examples.

[0040] In order to overcome the shortcomings of the prior art, the present invention provides a self-supervised graph reconstruction framework (SSGR-GT) based on Graph-Transformer to identify important brain regions. Compared with existing work, deep learning based on self-supervision does not rely on manual features and can learn the representation of the data itself. First, the MRI data of the brain is constructed into a multimodal brain map, and the structural information and functional information of the brain are mapped to the edge features and point features of the graph respectively; then, the important brain map nodes are identified through a self-supervised model (GR-GT). GR-GT adopts an encoder-decoder architecture, takes the brain map as input and outputs a reconstructed brain map, requires the reconstructed brain map to be as close to the original brain map as possible, and extracts the contribution scores of all brain map nodes to the reconstruction task. Nodes with high contribution scores are more important to the reconstruction task and are the important brain regions identified by the present invention.

[0041] like Figure 1 As shown in Figure 2, the proposed SSGR-GT framework includes the following steps:

[0042] Step 1: Construct a multimodal brain map using structural and functional brain data;

[0043] The brain is defined as an undirected graph G = {V, E}. V = {v i |i∈1,2,…,N} represents the node set of the graph, which has N nodes. Represents the node features of the graph. Represents the adjacency matrix of the graph. Figure 1 As shown in (A), the brain partition template is used to divide the brain surface into N regions as a set of nodes for the brain map. Then, the fMRI signals within the region are averaged as the functional signals of the nodes, and the Pearson correlation coefficient of the signals between each pair of nodes is calculated to obtain the functional similarity matrix. The functional similarity matrix is used as the node feature matrix X of the graph, and the feature vector length of each node is D = N. Finally, using DTI data, the total number of fibers directly connected to each pair of nodes is calculated and divided by the geometric mean of the node area to obtain the structural connection matrix. The structural connection matrix is thresholded to make it sparse and binarized to obtain the adjacency matrix E of the graph. E i，j =1 means that nodes i and j have an edge, otherwise E i，j =0.

[0044] Step 2: Obtain the contribution score of the brain node through GR-GT;

[0045] like Figure 1As shown in (B), in GR-GT, the present invention uses an encoder-decoder structure to implement the graph reconstruction task and obtain the contribution scores of nodes. First, the brain map obtained in step 1 is used as input. The encoder module extracts the node representation of the graph. Then, the mask score module obtains the node contribution score. Finally, the decoder module reconstructs the node features of the graph.

[0046] Encoder module: such as Figure 2 As shown in (A), the Transformer is combined with the GCN. First, the Transformer layer is used to apply the self-attention mechanism to all nodes in the graph. Then, the GCN is used to combine the topological information of the graph. The process is shown in Equations (4) and (5).

[0047] H′ (k+1) =Attention V (H (k) Q (k) , H (k) K (k) , H (k) V (k) ) (4)

[0048] H (k+1) =GCN(E,H′ (k+1) ) (5) Among them H (k) is the graph node representation of the kth layer in the encoder module, V is the set of all nodes in the graph, is the projection matrix, Attention V Indicates the use of self-attention mechanism for all nodes of the graph, and E represents the adjacency matrix of the graph. The detailed expression of formula (4) is:

[0049]

[0050] in is the representation vector of node v, Trans and Att represent MLP and multi-head attention respectively, and Lnorm represents normalization. The detailed expression of Att is:

[0051]

[0052] in is the attention weight between nodes v and v′, and the remaining variables have been mentioned in other formulas. The detailed expression of formula (5) is:

[0053]

[0054] Where I is the identity matrix, σ is the nonlinear function, W is the weight matrix, and the remaining variables have been mentioned in other formulas.

[0055] Mask score module: such as Figure 2 As shown in (A), this module is located after the encoder module and before the decoder module. The output of the encoder module is the node representation of the graph As the input of this module. After passing through the MLP layer and softmax, Z is mapped to p is considered the contribution score of the graph node. Nodes with lower scores are then masked according to a certain ratio k and assigned a mask representation vector. Finally, a Hadamard product is performed on the graph node representation and the contribution score, which serves as the input to the decoding module. The calculation process is shown in Equation (9).

[0056] p = softmax(MLP(Z))

[0057] idx = rank(p, k)

[0058]

[0059]

[0060] Where Z is the graph node representation output by the encoder module, softmax is a nonlinear function, p is the contribution score of the graph node, rank is the ranking function, idx is the index with the highest value after sorting p, k is the proportion of shielded nodes, q is the mask representation vector of the shielded nodes, cat is the concatenation of the node representation and the mask representation vector in the original order, ⊙ is the Hadamard product, and Z′ is the input of the decoder module.

[0061] Decoder module: such as Figure 2 As shown in (A), the decoder module takes as input the representation of graph nodes with mask information and outputs the reconstructed graph node features. The decoder module has the same structure as the encoder module. The GR-GT module reconstructs the graph by predicting the node features. The loss function is the mean square error (MSE) between the feature matrix Y of the reconstructed graph nodes and the feature matrix X of the input graph nodes.

[0062] Step 3: Pass the dataset through steps 1 and 2 to obtain the contribution scores of the reconstructed graph and nodes. The areas with high scores are the final important brain areas.

[0063] The present invention uses multiple fMRI data sets from public datasets for experiments, including one resting-state fMRI and seven task-based fMRIs. For all fMRI data, corresponding multimodal brain maps combined with structure were obtained through step 1. The present invention set up eight groups of experiments, each of which was trained independently. Each group of experiments used multimodal brain maps corresponding to the same type of fMRI for all individuals. For each group of experiments, important areas of the corresponding multimodal brain network were obtained. Conducting multiple groups of experiments is conducive to a detailed analysis of the integration of brain structure and function.

[0064] During model training, the number of layers in the encoder and decoder modules was set to 1, the number of heads in the multi-head attention was set to 4, the number of epochs was set to 200, and the Adam optimizer was used to update the parameters. The node mask ratio k was set to 0.5.

[0065] To extract key brain regions, the model provides a regional score for each individual. Statistical analysis of all individual regional scores is required, using methods such as averages to derive group-level regional scores. Once the group scores are obtained, brain regions are ranked based on the scores, and the regions with the highest scores are selected as key brain regions according to a specific ratio. Specific embodiment:

[0067] 1) Constructing multimodal brain maps using structural and functional data of the brain;

[0068] The brain is defined as an undirected graph G = {V, E}. V = {v i |i∈1,2,…,N} represents the node set of the graph, which has N nodes. Represents the node features of the graph. Represents the adjacency matrix of the graph. Figure 1 As shown in (A), the brain partition template is used to divide the brain surface into N regions as a set of nodes for the brain map. Then, the fMRI signals within the region are averaged as the functional signals of the nodes, and the Pearson correlation coefficient of the signals between each pair of nodes is calculated to obtain the functional similarity matrix. The functional similarity matrix is used as the node feature matrix X of the graph, and the feature vector length of each node is D = N. Finally, using DTI data, the total number of fibers directly connected to each pair of nodes is calculated and divided by the geometric mean of the node area to obtain the structural connection matrix. The structural connection matrix is thresholded to make it sparse and binarized to obtain the adjacency matrix E of the graph. E i，j =1 means that nodes i and j have an edge, otherwise E i，j =0.

[0069] 2) Obtain the contribution score of brain nodes through GR-GT;

[0070] like Figure 1 As shown in (B), in GR-GT, the present invention uses an encoder-decoder structure to implement the graph reconstruction task and obtain the contribution scores of nodes. First, the brain map obtained in step 1 is used as input. The encoder module extracts the node representation of the graph. Then, the mask score module obtains the node contribution score. Finally, the decoder module reconstructs the node features of the graph.

[0071] Encoder module: such as Figure 2 As shown in (A), the Transformer is combined with the GCN. First, the Transformer layer is used to apply the self-attention mechanism to all nodes in the graph. Then, the GCN is used to combine the topological information of the graph. The process is shown in Equations (10) and (11).

[0072] H′ (k+1) =Attention V (H (k) Q (k) , H (k) K (k) , H (k) V (k) ) (10)

[0073] H (k+1) =GCN(E,H′ (k+1) ) (11) where H (k) is the graph node representation of the kth layer in the encoder module, V is the set of all nodes in the graph, is the projection matrix, Attention V Indicates the use of self-attention mechanism for all nodes of the graph, and E represents the adjacency matrix of the graph. The detailed expression of formula (10) is:

[0074]

[0075] in is the representation vector of node v, Trans and Att represent MLP and multi-head attention respectively, and Lnorm represents normalization. The detailed expression of Att is:

[0076]

[0077] in is the attention weight between nodes v and v′, and the remaining variables have been mentioned in other formulas. The detailed expression of formula (11) is:

[0078]

[0079] Where I is the identity matrix, σ is the nonlinear function, W is the weight matrix, and the remaining variables have been mentioned in other formulas.

[0080] Mask score module: such as Figure 2 As shown in (A), this module is located after the encoder module and before the decoder module. The output of the encoder module is the node representation of the graph As the input of this module. After passing through the MLP layer and softmax, Z is mapped to p is considered the contribution score of the graph node. Nodes with lower scores are then masked according to a certain ratio k and assigned a mask representation vector. Finally, a Hadamard product is performed on the graph node representation and the contribution score, which serves as the input to the decoding module. The calculation process is shown in Equation (15).

[0081] p = softmax(MLP(Z))

[0082] idx = rank(p, k)

[0083]

[0084]

[0085] Where Z is the graph node representation output by the encoder module, softmax is a nonlinear function, p is the contribution score of the graph node, rank is the ranking function, idx is the index with the highest value after sorting p, k is the proportion of shielded nodes, q is the mask representation vector of the shielded nodes, cat is the concatenation of the node representation and the mask representation vector in the original order, ⊙ is the Hadamard product, and Z′ is the input of the decoder module.

[0086] Decoder module: such as Figure 2 As shown in (A), the decoder module takes as input the representation of graph nodes with mask information and outputs the reconstructed graph node features. The decoder module has the same structure as the encoder module. The GR-GT module reconstructs the graph by predicting the node features. The loss function is the mean square error (MSE) between the feature matrix Y of the reconstructed graph nodes and the feature matrix X of the input graph nodes.

[0087] 3) Model testing, obtaining the reconstructed graph and node contribution scores. The areas with high scores are the final important brain areas.

[0088] The present invention employed a variety of fMRI data for experiments, including one resting-state fMRI and seven task-based fMRI data. For all fMRI data, corresponding multimodal brain maps were obtained through step 1, along with the structure. Eight experimental groups were set up, each trained independently. Each group used multimodal brain maps corresponding to the same fMRI type from all individuals, and obtained key regions of the corresponding multimodal brain network. Regarding the model's graph reconstruction performance, the experimental loss value decreased with epoch, ultimately becoming relatively stable. Furthermore, to verify the effectiveness of the reconstructed graph, the cosine similarity and structural similarity between the node feature matrices of the original brain map and the reconstructed brain map were calculated. Both values increased with epoch, ultimately reaching individual averages of 0.86 and 0.82, respectively, demonstrating the model's good reconstruction performance. Regarding the node scores of the brain maps, node scores fluctuated significantly with fewer epochs, but gradually stabilized with more epochs, ultimately enabling clear distinction between the scores of all nodes in the brain map.

[0089] The present invention proposes a self-supervised graph reconstruction framework based on Graph-Transformer, which identifies important brain regions by fusing multimodal neuroimaging data. It includes: (1) through self-supervised deep learning, some regions that contribute greatly to the reconstruction of brain maps are regarded as important brain regions; (2) the model built on Graph-Transformer combines the local and global connection information of the graph to well extract brain map features; (3) the fusion of brain structure and function data can fully understand how the brain network architecture is generated from multimodal connections. The present invention conducts experiments on the integration of structure and different functional data, obtains important brain regions under different multimodal conditions, and finds that there are some common regions between them. The method proposed by the present invention analyzes data in a completely data-driven way rather than manual features, which effectively avoids the inherent limitations brought by prior knowledge. Through verification, it is found that compared with the existing method based on graph theory prior knowledge to obtain brain centers as important brain regions, the important brain regions identified by the present invention have a large overlap with brain centers, and some new regions are discovered as important brain regions. Furthermore, to verify the stability of the key brain regions extracted from the model, the present invention tested the model on a new brain parcellation atlas, demonstrating good stability. In summary, this invention explores a new method for exploring key brain regions using self-supervised learning, providing a new approach to explaining how the brain works and helping to understand its functions of information transmission, separation, and integration.

Claims

1. A multimodal brain network important region identification method based on self-supervised Graph-Transformer, characterized by: The steps include: Step 1: Construct a multimodal brain map from brain MRI data; The brain is defined as an undirected graph G = {V, E}, V = {v i |i∈1,2,…,N} represents the node set of the graph, which has N nodes; Represents the node features of the graph; Represents the adjacency matrix of the graph; First, the brain surface is divided into N regions using a brain partition template, which serve as nodes of the graph. Then, using functional MRI signals, the Pearson correlation coefficient of the functional signals between nodes is calculated to obtain a functional similarity matrix, which is used as the node feature matrix X of the graph. The length of the eigenvector of each node is D = N. Finally, using diffusion tensor imaging (DTI) data, the fiber connections between nodes are calculated to obtain a structural connectivity matrix, which is then sparsified and binarized to obtain the graph adjacency matrix E. Step 2: Obtain the contribution score of the brain node through the self-supervised model GR-GT module; In the self-supervised model GR-GT module, an encoder-decoder structure is used to implement the graph reconstruction task and obtain the contribution score of the node; First, the brain map obtained in step 1 is used as input, and the encoder module is used to extract the node representation of the graph. Then, the mask score module is used to obtain the contribution score of the node, and finally the decoder module is used to reconstruct the node features of the graph. Encoder module: Combine Transformer with graph convolutional neural network (GCN). First, the Transformer layer is used to apply the self-attention mechanism to all nodes of the graph. Then, GCN is used to combine the topological information of the graph. The process is shown in Equations (1) and (2): H' (k+1) =Attention v (H (k) Q (k) ,H (k) K (k) ,H (k) V (k) ) (1) H (k+1) =GCN(E,H' (k+1) ) (2) Among them H (k) is the graph node representation of the kth layer in the encoder module, V is the set of all nodes in the graph, is the projection matrix, Attention V Indicates the use of self-attention mechanism for all nodes of the graph, and E represents the adjacency matrix of the graph; Mask score module: This module is placed after the encoder module and before the decoder module; the output of the encoder module is the node representation of the graph As the input of the mask score module; after passing through the multi-layer perceptron MLP and softmax, Z is mapped to p is regarded as the contribution score of the graph node; then, according to the ratio k, the nodes with lower scores are masked and assigned a mask representation vector; finally, the Hadamard product operation is performed on the node representation of the graph and the contribution score, which is used as the input of the decoding module; the calculation process is shown in formula (3): Where softmax is a nonlinear function, rank represents the ranking function, idx represents the index with the highest value after sorting p, q represents the mask representation vector of the masked node, cat represents the concatenation of the node representation and the mask representation vector in the original order, ⊙ represents the Hadamard product, and Z′ represents the input of the decoder module; Decoder module: The input of the decoder module is the graph node representation with mask information, and the output is the node features of the reconstructed graph; The decoder module has the same structure as the encoder module; The GR-GT module reconstructs the graph by predicting the node features of the graph; the loss function is the mean square error (MSE) between the feature matrix Y of the reconstructed graph nodes and the feature matrix X of the input graph nodes; Step 3: The dataset is passed through steps 1 and 2 to obtain the contribution scores of the reconstructed graph and nodes. The brain regions are ranked according to the scores, and the regions with the highest scores are selected as important brain regions according to the set ratio.

2. A multimodal brain network important region identification method based on self-supervised Graph-Transformer according to claim 1, characterized in that: The detailed expression of formula (4) is: in is the representation vector of node v, Trans and Att represent MLP and multi-head attention respectively, and Lnorm represents normalization; The detailed expression of Att is: in is the attention weight between nodes v and v′.

3. The method for identifying important regions of a multimodal brain network based on self-supervised Graph-Transformer according to claim 2 is characterized in that: The detailed expression of formula (5) is: Where I is the identity matrix, σ is the nonlinear function, and W is the weight matrix.

4. The method for identifying important regions of a multimodal brain network based on self-supervised Graph-Transformer according to claim 1, characterized in that: During training, the GR-GT module sets the number of layers in the encoder and decoder modules to 1, the number of heads in the multi-head attention to 4, the number of epochs to 200, and uses the Adam optimizer to update parameters, with the node mask ratio k being 0.5.

Images