A Joint Learning Method for Brain Network Structure and Similarity Based on Graph Attention Network

Abstract

The present invention belongs to the fields of deep learning and brain network structure, and particularly relates to a joint learning method for brain network structure and similarity based on graph attention network, including: performing cortical segmentation processing and morphological feature extraction on the obtained brain image data, and modeling the subject's brain network as a graph; estimating the initial brain network structure through Pearson correlation calculation; obtaining the similarity between brain network structures through a siamese graph attention learning network; calculating the graph regularization loss function and the siamese network loss function to constrain the characteristics of the initial brain network structure; updating the adjacency matrix of the brain network according to the embedding features of the brain network, and obtaining the updated brain network structure and calculating the similarity of the brain network structure. The present invention effectively estimates the morphological brain network by jointly optimizing the two tasks of brain network structure estimation and similarity learning, and provides valuable information for subsequent tasks such as individual recognition and disease auxiliary diagnosis.

Description

Technical Field

[0001] The present invention belongs to the fields of deep learning and brain network structure, and particularly relates to a joint learning method for brain network structure and similarity based on a graph attention network. Background Art

[0002] Alzheimer's disease (AD, commonly known as senile dementia) is one of the most common neurodegenerative diseases in the elderly population. The clinical features of AD are the decline of memory and other cognitive functions, and there is currently no effective treatment method. Mild cognitive impairment (MCI), as a transitional stage between normal aging and AD, its accurate diagnosis is very important for the early treatment of AD and delaying the disease process. The development of computer-aided diagnosis technology based on medical images for brain diseases such as MCI has become a research hotspot in the field of neuroscience today. However, due to the mild symptoms in the MCI stage and the not obvious changes in brain function and anatomical structure, accurate auxiliary diagnosis of MCI diseases still has quite a big challenge.

[0003] Constructing an individual morphological brain network based on sMRI for a single subject can provide important evidence for understanding the whole-brain morphological connection pattern in disease states such as MCI, and provide sufficiently sensitive and specific imaging markers for clinical applications such as disease auxiliary diagnosis and efficacy evaluation. However, the current methods for constructing individual morphological brain networks are not yet mature. On the one hand, they are limited by the speed and accuracy of cerebral cortex surface segmentation, and on the other hand, they usually do not consider subsequent tasks such as disease diagnosis, and their biological significance needs to be improved. Therefore, researching and establishing a reliable method for constructing individual morphological brain networks, revealing the abnormal mechanism of MCI brain networks on this basis, and improving the accuracy of auxiliary diagnosis of this disease are an urgent need in the current research on MCI diseases, and have important academic significance and clinical value.

[0004] Currently, most of the existing methods for constructing individual morphological brain networks are only based on one morphological feature, and estimate the brain network structure by calculating the similarity of the average value of features between ROIs or the statistical correlation of eigenvalue distributions, which cannot fully reflect the complex characteristics of the cerebral cortex. In recent years, researchers have proposed to estimate the brain network structure using multiple morphological features, improving the ability of auxiliary diagnosis and prediction of diseases. According to the different calculation methods of morphological connections between nodes, these algorithms can be roughly divided into methods based on pairwise ROIs and methods based on multiple ROIs.

[0005] The morphological connections estimated by the pairwise ROI-based method reflect the correlation between pairwise ROIs. Pearson correlation analysis is the most commonly used method among them, which directly captures the similarity of the eigenvectors of node pairs. Professor Lihua Li of Hangzhou Dianzi University and others calculated the "higher-order" morphological similarity between ROIs using various feature distances, providing high-level supplementary information for the traditional Pearson correlation method. They proposed to measure the correlation of feature distributions between ROIs by calculating the exponential function of the multivariate Euclidean distance, effectively estimating the individual morphological brain network structure and improving the accuracy of disease auxiliary diagnosis. Although these methods are simple to calculate and have biological intuitiveness, they ignore the potential influence of other regions and may produce incorrect connections. To address this problem, the multiple ROI-based method considers the influence of multiple regions and usually estimates the brain network structure by solving an optimization problem with an L1-norm regularization term, introducing a sparse prior for the network structure to make it more biologically meaningful. Although such methods have been rarely reported in the research of morphological brain networks, in the parallel field of functional brain network construction, it has become a research trend in recent years to construct a more reasonable brain network structure by improving the sparse regularization term or introducing a new regularization term.

[0006] However, the existing brain network construction is prior to any brain network analysis, and the obtained brain network structure may not be optimal for subsequent tasks such as individual recognition and disease auxiliary diagnosis. To address this problem, in the related field of graph signal processing (the brain network can be regarded as a graph), researchers have proposed to use the graph neural network technology in geometric deep learning to adaptively adjust the graph structure for downstream tasks. These pioneering studies provide important ideas for designing the estimation method of brain network structure, but they are not oriented to the design of brain network structure and do not use the information that has been proven valuable in brain network construction, such as the sparsity and modularity of the brain network.

[0007] In summary, the existing technical problems are as follows:

[0008] 1. The current individual morphological brain network construction method is not yet mature. On the one hand, it is limited by the speed and accuracy of cerebral cortex surface segmentation, and on the other hand, it usually does not consider subsequent tasks such as disease diagnosis, and its biological significance needs to be improved;

[0009] 2. By calculating the exponential function of the multivariate Euclidean distance to measure the correlation of feature distributions between ROIs, these methods are simple to calculate and have biological intuitiveness, but they ignore the potential influence of other regions and may produce incorrect connections;

[0010] 3. The existing brain network structure estimation does not propose an adaptive brain network structure estimation method for subsequent tasks. Summary of the Invention

[0011] To solve the above technical problems, the present invention proposes a method for learning the relationship between brain network structure and similarity based on graph attention network, including the following steps:

[0012] S1: Obtain the sMRI data of the subject's brain, perform cortical segmentation processing and morphological feature extraction on the obtained sMRI data of the brain to obtain the morphological features of all brain regions, model the subject's brain network as a graph, each node in the brain network corresponds to a brain region, and form the feature matrix of the brain network according to the morphological features of each brain region node;

[0013] S2: Map the feature matrix of the brain network to the adjacency matrix of the graph through Pearson correlation calculation, estimate the initial brain network structure, and set a threshold to remove weak connections;

[0014] S3: Input the feature matrix and adjacency matrix of a pair of brain networks after removing weak connections into the siamese graph attention learning network, perform graph embedding representation learning through the feature matrix of the brain network to obtain the embedded feature representation, and calculate the similarity between brain network structures according to the embedded feature representation and adjacency matrix of a pair of brain networks;

[0015] S4: Calculate the graph regularization loss function and the siamese network loss function to constrain the sparsity, modularity, intra-group similarity and inter-group dissimilarity of the initial brain network structure;

[0016] S5: Update the adjacency matrix of the brain network according to the embedded features of the brain network, and combine it with the initial brain network structure that constrains sparsity, modularity, intra-group similarity and inter-group dissimilarity to obtain the updated brain network structure;

[0017] S6: Learn new embedded feature representations according to the feature matrix of the updated brain network structure, and calculate the similarity of the brain network structure according to the calculated new embedded feature representations.

[0018] Preferably, the subject's brain network is modeled as a graph g =

[0019] {v, ε, x, A}, where is a set of n brain network nodes, each node corresponds to a brain region, and ε is the set of brain network edges; is the feature matrix of the node, and the feature vector of each node is the average of the morphological feature vectors of all vertices in the corresponding brain region, and d is the dimension of the node features; is the adjacency matrix of the graph, representing the connection relationship between nodes.

[0020] Preferably, map the feature matrix of the brain network to the adjacency matrix of the graph through Pearson correlation calculation, and estimate the initial brain network structure, which is expressed as:

[0021]

[0022] Among them, a ij represents the estimated morphological brain network structure, represents the cross-covariance, σ(·) represents the sigmoid function, and ⊙ represents the Hadamrd product, represents the trainable weight vector, represents the feature matrix of the i-th brain region node in the cerebral cortex, represents the feature matrix of the j-th brain region node in the cerebral cortex.

[0023] Preferably, a threshold is set to remove weak connections, which is expressed as:

[0024]

[0025] Among them, represents the morphological brain network structure after removing weak connections, and a ij represents the estimated morphological brain network structure, and ξ represents the threshold.

[0026] Preferably, graph embedding representation learning is performed through the feature matrix of the brain network to obtain the embedded feature representation, which is expressed as:

[0027]

[0028] Among them, is the currently learned embedded feature representation, represents concatenating H attention heads, σ is the sigmoid function, and v j represents the node in the h-th attention head, and v i represents the node in the first-order neighborhood within the h-th attention head, represents the number of nodes within the attention head, represents the attention weight of the h-th head, represents a trainable linear transformation matrix, and x j represents the feature matrix of the j-th brain region node in the brain network.

[0029] Preferably, the similarity between brain network structures is calculated based on the embedded feature representation and the adjacency matrix of a pair of brain networks, which is expressed as:

[0030]

[0031] Among them, O p represents the similarity score, Att(·) represents the graph attention layer, FC(·) represents the fully connected layer, and <·,·> represents the inner product, represents an embedded representation of the brain network, represents an adjacency matrix of the brain network.

[0032] Preferably, the graph regularization loss function is calculated and expressed as:

[0033]

[0034] Wherein, represents the regularization loss function, represents the optimized brain network structure, ‖·‖ 1 represents the sparse loss, ‖·‖ * represents the low-rank loss, represents two subjects in the brain network structure pair, λ 1 and λ 2 represent hyperparameters for adjusting the weights between loss terms.

[0035] Preferably, the siamese network loss function is calculated and expressed as:

[0036]

[0037] Wherein, represents the Hinge loss function, N p is the number of brain network structure pairs with high similarity; Y p is the true similarity label of the brain network structure pair, O p is the similarity score output by the siamese network.

[0038] Preferably, the S5 is expressed as

[0039]

[0040] Wherein, represents the updated brain network structure, represents the initial brain network structure, represents the updated adjacency matrix, and β represents the weight.

[0041] Advantages of the present invention: By using a trainable Pearson correlation to estimate the brain network structure, a siamese graph network is used to learn the similarity between brain networks, and the brain network structure and graph embedding representation are iteratively updated to jointly optimize the two tasks of brain network structure estimation and similarity learning, effectively estimating the morphological brain network and providing valuable information for subsequent tasks such as individual recognition and disease assistant diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flowchart of the present invention;

[0043] Figure 2 is a schematic diagram of the relationship learning between the brain network structure and similarity of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0044] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0045] A method for learning the relationship between the brain network structure and similarity based on the graph attention network, as Figure 2 shown, includes the following steps, as Figure 1 shown:

[0046] S1: Obtain the sMRI data of the subject's brain, perform cortical segmentation processing and morphological feature extraction on the obtained brain sMRI data to obtain the morphological features of all brain regions, and model the subject's brain network as a graph. Each node in the brain network corresponds to a brain region, and the feature matrix of the brain network is composed according to the morphological features of each brain region node;

[0047] S2: Map the feature matrix of the brain network to the adjacency matrix of the graph through Pearson correlation calculation, estimate the initial brain network structure, and set a threshold to remove weak connections;

[0048] S3: Input the feature matrix and adjacency matrix of a pair of brain networks after removing weak connections into the Siamese graph attention learning network, perform graph embedding representation learning through the feature matrix of the brain network to obtain the embedded feature representation, and calculate the similarity between brain network structures according to the embedded feature representation and adjacency matrix of a pair of brain networks;

[0049] S4: Calculate the graph regularization loss function and the Siamese network loss function to constrain the sparsity, modularity, intra-group similarity, and inter-group dissimilarity of the initial brain network structure;

[0050] S5: Update the adjacency matrix of the brain network according to the embedded features of the brain network, and combine it with the initial brain network structure that constrains sparsity, modularity, intra-group similarity, and inter-group dissimilarity to obtain the updated brain network structure;

[0051] S6: Learn new embedded feature representations according to the feature matrix of the updated brain network structure, and calculate the similarity of the brain network structure according to the calculated new embedded feature representations.

[0052] Model the subject's brain network as a graph g = {v, ε, x, A} through cerebral cortex segmentation processing, where is a set of n brain network nodes, each node corresponds to a brain region, and ε is the set of brain network edges; is the feature matrix of the nodes, and the feature vector of each node is the average of the morphological feature vectors of all vertices in the corresponding brain region. d is the dimension of the node features; is the adjacency matrix of the graph, representing the connection relationship between nodes.

[0053] The cerebral cortex segmentation method: Obtain the brain sMRI image, reconstruct the cerebral cortex surface, and extract the morphological features of the cerebral cortex surface; the brain sMRI image includes the target image to be segmented and the atlas set image that has been segmented; Use graph Laplacian to perform feature decomposition on the multi-dimensional morphological features of the cerebral cortex surface to obtain the spectral representation of the cerebral cortex surface manifold; Use the segmentation module with a U-shaped hierarchical structure to extract the context information of the cerebral cortex surface of the target image, and use the auxiliary module with the same U-shaped hierarchical structure to extract the anatomical prior information of the cerebral cortex surface of the atlas set image, and use the non-local attention feature fusion module to fuse the information of the segmentation module and the information of the auxiliary module, and output the prediction probability map of the segmentation label of the cerebral cortex surface of the target image.

[0054] The reconstruction of the cerebral cortex surface includes mapping the brain sMRI image to the standard space, using the N3 algorithm to correct the inhomogeneity of the brain sMRI image, removing non-brain tissues, and then segmenting the brain white matter, brain gray matter, and cerebrospinal fluid based on intensity values and neighborhood constraints; Reconstruct the inner surface of the cerebral cortex using the interface between the brain white matter and the brain gray matter, and generate the outer surface of the cerebral cortex using the interface between the brain gray matter and the cerebrospinal fluid.

[0055] The extraction of the morphological features of the cerebral cortex surface includes, for each vertex on the inner surface of the cerebral cortex, extracting five morphological features that reflect different geometric attributes of the cerebral cortex, including: Mean curvature, whose value is the reciprocal of the radius of the inscribed sphere at the vertex; Gaussian curvature, whose value is the product of the principal curvatures at the vertex, reflecting the degree of curvature of the surface in different directions at that point; Cortical thickness, whose value is the distance between the corresponding vertices of the white matter surface and the gray matter surface; Sulcus depth, whose value is the vertical distance from the vertex to the middle surface of the gray and white matter; Surface area, whose value is the average area of all adjacent triangular patches of the vertex.

[0056] The segmentation module of the U-shaped hierarchical structure includes a first encoding layer and a first decoding layer; the first encoding layer includes a first input layer, and a plurality of first feature extraction layers, first pooling layers, and first non-local attention feature fusion modules arranged in repetition; the first decoding layer includes a plurality of second feature extraction layers, a first upsampling layer, and a second non-local attention feature fusion module arranged in repetition, and a first output layer; the auxiliary module of the U-shaped hierarchical structure includes a second encoding layer and a second decoding layer; the second encoding layer includes a second input layer, and a plurality of third feature extraction layers and second pooling layers arranged in repetition; the second decoding layer includes a plurality of fourth feature extraction layers and a second upsampling layer; wherein, the first pooling layer is connected to the first feature extraction layer and the third feature extraction layer, and the second pooling layer is connected to the second feature extraction layer and the fourth feature extraction layer; the first non-local attention feature fusion module is connected to the first pooling layer and the second pooling layer, and the second non-local attention feature fusion module is connected to the first upsampling layer and the second upsampling layer.

[0057] The feature matrix of the brain network is mapped to the adjacency matrix of the graph through Pearson correlation calculation, and the initial brain network structure is estimated, expressed as:

[0058]

[0059] where a ij represents the estimated morphological brain network structure, represents the cross-covariance, σ(·) represents the sigmoid function, ⊙ represents the Hadamrd product, and w s represents the trainable weight vector, represents the feature matrix of the i-th brain region node in the cerebral cortex, represents the feature matrix of the j-th brain region node in the cerebral cortex.

[0060] Set a threshold to remove weak connections, expressed as:

[0061]

[0062] where, represents the morphological brain network structure after removing weak connections, a ij represents the estimated morphological brain network structure, and ξ represents the threshold.

[0063] For the learning of graph embedding representation, a multi-head graph attention mechanism is proposed. Graph attention is a space-based graph convolution method suitable for heterogeneous graphs. First, calculate the attention weights of the h-th head:

[0064]

[0065] where, Denote the attention weight of the h-th head, Denote a trainable linear transformation matrix, ‖ denotes the concatenation operation, Denote a trainable weight vector, ELU denotes the exponential linear unit activation function, T denotes the transpose operation, x i Denote the feature matrix of the i-th brain region node of the brain network, x j Denote the feature matrix of the j-th brain region node of the brain network, v j Denote the node within the h-th attention head, v i Denote the nodes within the first-order neighborhood of the h-th attention head, Denote the number of nodes within the attention head.

[0066] Then aggregate all the neighborhood information of the node v i and concatenate all H attention heads to obtain its embedded feature representation:

[0067]

[0068] where, The currently learned embedded feature representation, Denote the concatenation of H attention heads, σ is the sigmoid function, v j Denote the node within the h-th attention head, v i Denote the nodes within the first-order neighborhood of the h-th attention head, Denote the number of nodes within the attention head, Denote the attention weight of the h-th head, Denote a trainable linear transformation matrix, x j Denote the feature matrix of the j-th brain region node of the brain network.

[0069] Learn the similarity between brain network structures based on the embedded feature representations and adjacency matrices of a pair of brain networks, denoted as:

[0070]

[0071] where, O p Denote the similarity score, Att(·) denotes the graph attention layer, FC(·) denotes the fully connected layer, <·,·> denotes the inner product, Denote an embedded representation of the brain network, Denote an adjacency matrix of the brain network.

[0072] Calculate the graph regularization loss function, denoted as:

[0073]

[0074] where, denotes the regularization loss function, denotes the optimized brain network structure, ‖·‖ 1 denotes the sparse loss, ‖·‖ * denotes the low-rank loss, denotes two subjects in the brain network structure pair, λ 1 and λ 2 denote the hyperparameters used to adjust the weights between loss terms.

[0075] Calculate the loss function of the siamese network, denoted as:

[0076]

[0077] where, denotes the Hinge loss function, N p is the number of brain network structure pairs with high similarity; Y p is the true similarity label of the brain network structure pair, O p is the similarity score output by the siamese network.

[0078] The so-called S5, denoted as

[0079]

[0080] where, denotes the updated brain network structure, denotes the initial brain network structure, denotes the updated adjacency matrix, and β represents the weight.

[0081] Learn a new embedded feature representation based on the feature matrix of the updated brain network structure:

[0082]

[0083] where, denotes the newly learned embedded feature representation, Att(·) represents the graph attention layer, denotes the feature matrix of the updated brain network structure, denotes an adjacency matrix of the updated brain network structure;

[0084] Calculate the similarity of the brain network structure based on the calculated new embedded feature representation:

[0085]

[0086] where, denotes an embedded representation of the updated brain network structure, An adjacency matrix representing the updated brain network structure, Att(·) represents the graph attention layer, FC(·) represents the fully connected layer, <·,·> represents the inner product, and O p is the similarity score.

[0087] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for learning the relationship between brain network structure and similarity based on graph attention network, characterized in that, it includes the following steps: S1: Obtain the sMRI data of the subject's brain, perform cortical segmentation processing and morphological feature extraction on the obtained brain sMRI data to obtain the morphological features of all brain regions, and model the subject's brain network as a graph. Each node in the brain network corresponds to a brain region, and a feature matrix of the brain network is formed according to the morphological features of each brain region node; S2: Map the feature matrix of the brain network to the adjacency matrix of the graph through Pearson correlation calculation, estimate the initial brain network structure, and set a threshold to remove weak connections; S3: Input the feature matrices and adjacency matrices of a pair of brain networks after removing weak connections into the twin graph attention learning network, perform graph embedding representation learning through the feature matrix of the brain network to obtain the embedded feature representation, and calculate the similarity between brain network structures according to the embedded feature representations and adjacency matrices of a pair of brain networks; S4: Calculate the graph regularization loss function and the twin network loss function to constrain the sparsity, modularity, intra-group similarity, and inter-group dissimilarity of the initial brain network structure; S5: Update the adjacency matrix of the brain network according to the embedded features of the brain network, and combine it with the initial brain network structure that constrains sparsity, modularity, intra-group similarity, and inter-group dissimilarity to obtain the updated brain network structure; S6: Learn a new embedded feature representation according to the feature matrix of the updated brain network structure, and calculate the similarity of the brain network structure according to the newly learned embedded feature representation.

2. The method for learning the relationship between brain network structure and similarity based on graph attention network according to claim 1, characterized in that, The brain network of the subject is modeled as a graph \(g = \{v, \varepsilon, x, A\}\) through cerebral cortex segmentation processing, where is a set of \(n\) brain network nodes, each node corresponding to a brain region, and \(\varepsilon\) is a set of brain network edges; is the feature matrix of the nodes, and the feature vector of each node is the average of the morphological feature vectors of all vertices in the corresponding brain region. \(d\) is the dimension of the node features; is the adjacency matrix of the graph, representing the connection relationship between nodes.

3. The method for learning the relationship between brain network structure and similarity based on graph attention network according to claim 1, characterized in that, Map the feature matrix of the brain network to the adjacency matrix of the graph through Pearson correlation calculation, and estimate the initial brain network structure, expressed as: Among them, a ij represents the estimated morphological brain network structure, represents the cross-covariance, σ(·) represents the sigmoid function, and ⊙ represents the Hadamrd product, represents the trainable weight vector, represents the feature matrix of the i-th brain region node in the cerebral cortex, represents the feature matrix of the j-th brain region node in the cerebral cortex.

4. The method for learning the relationship between brain network structure and similarity based on graph attention network according to claim 1, characterized in that, Set a threshold to remove weak connections, expressed as: Among them, represents the morphological brain network structure after removing weak connections, and a ij represents the estimated morphological brain network structure, and ξ represents the threshold.

5. The method for learning the relationship between brain network structure and similarity based on graph attention network according to claim 1, characterized in that, Perform graph embedding representation learning through the feature matrix of the brain network to obtain the embedded feature representation, expressed as: Among them, The currently learned embedded feature representation, Indicates concatenating H attention heads, σ is the sigmoid function, v j Represents the node within the h-th attention head, v i Represents the node within the first-order neighborhood of the h-th attention head, Indicates the number of nodes within the attention head, Represents the attention weight of the h-th head, Represents a trainable linear transformation matrix, x j Represents the feature matrix of the j-th brain region node in the brain network.

6. The method for learning the relationship between brain network structure and similarity based on graph attention network according to claim 1, characterized in that, Calculate the similarity between brain network structures according to the embedded feature representations and adjacency matrices of a pair of brain networks, expressed as: Among them, O p represents the similarity score, Att(·) represents the graph attention layer, FC(·) represents the fully connected layer, <·,·> represents the inner product, represents an embedded representation of the brain network, represents an adjacency matrix of the brain network.

7. The method for learning the relationship between brain network structure and similarity based on graph attention network according to claim 1, characterized in that, Calculate the graph regularization loss function, expressed as: Among them, represents the regularization loss function, represents the optimized brain network structure, ‖·‖ 1 represents the sparsity loss, ‖·‖ * represents the low-rank loss, represents two subjects in the brain network structure pair, λ 1 and λ 2 represent hyperparameters for adjusting the weights between the loss terms.

8. The method for learning the relationship between brain network structure and similarity based on graph attention network according to claim 1, characterized in that, Calculate the twin network loss function, expressed as: Among them, represents the Hinge loss function, and N p represents the number of pairs of brain network structures with high similarity; Y p represents the true similarity label of the pair of brain network structures, and O p represents the similarity score.

9. A learning method for the relationship between a brain network structure and similarity based on a graph attention network according to claim 1, characterized in that, the S5 is represented as Among them, represents the updated brain network structure, represents the initial brain network structure, represents the updated adjacency matrix, and β represents the weight.

Images