A method for extracting large-scale brain network features from multimodal magnetic resonance imaging

Abstract

The present invention discloses a method for extracting large-scale brain network features from multimodal magnetic resonance imaging (MRI), belonging to the field of medical image processing technology. Based on medical MRI data, this method obtains the brain's structural connectivity matrix through a series of preprocessing operations, extracts structural and functional features, and integrates functional MRI, structural MRI, and diffusion tensor imaging information. Structural connectivity is enhanced using a Node2vec graph embedding method, and features of brain region nodes are learned using branch attention and graph feature constraints. This solution effectively solves the problem of multimodal information fusion for brain diseases, overcomes the shortcomings of using diffusion tensor imaging to construct a structural connectivity matrix, and enables the learning of significant brain regions and important features related to brain diseases.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a method for extracting large-scale brain network features using multimodal magnetic resonance imaging. Background Art

[0002] In brain research, the widespread application of magnetic resonance imaging (MRI) technology provides an important research foundation. Different types of MRI, such as functional MRI (fMRI), structural MRI (sMRI), and diffusion tensor imaging (DTI), can describe brain characteristics from different perspectives.

[0003] Traditional brain connectome analysis uses magnetic resonance imaging data to explore disease characteristics from the perspective of connection topology, ignoring the properties of the nodes (brain regions) themselves. In addition, the research mainly focuses on a single brain region and single modality data, and lacks research on multimodal data fusion learning.

[0004] In recent years, deep graph neural networks (GNNs) have been proposed, which can learn embedded representations of nodes and extract high-order features of nodes. This makes it possible to explore the topological structure of brain diseases more comprehensively. For example, Chen et al. used the sparse brain maps of the subjects' fMRI models as input and proposed a novel low-parameter structural feature combination graph neural network (SFC-GNN) to select brain regions important for classification, thereby locating all active areas related to brain diseases. Zheng et al. proposed a new interpretable GNN framework BPI-GNN, which uses the well-known prototype learning to analyze fMRI. These studies use graph neural networks and their variants to achieve the fusion of brain networks and features.

[0005] However, these approaches also have certain limitations;

[0006] (1) In studies using magnetic resonance imaging to explore abnormal brain network organization, different modalities of magnetic resonance imaging can provide different perspectives to describe the characteristics of the brain, but current research has not yet fully integrated multimodal brain imaging data.

[0007] (2) When using graph neural networks to construct edge information of brain networks, the definition of graph edges is not accurate enough because functional connections are time-dependent and have negative connections, while white matter fiber connections based on DTI lack the stability to construct long-range connections.

[0008] (3) When there are many multimodal features, it is more accurate to identify features that are sensitive to disease detection. Summary of the Invention

[0009] In response to the above-mentioned deficiencies in the prior art, the multimodal magnetic resonance large-scale brain network feature extraction method provided by the present invention solves the above-mentioned technical problems. This method can fuse multimodal brain imaging data in the context of medical magnetic resonance brain imaging data analysis, construct more accurate brain structure connections, focus on learning significant brain regions and important features related to diseases, and ultimately use it to achieve classification tasks related to abnormal brain tissue through model training.

[0010] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a multimodal magnetic resonance large-scale brain network feature extraction method, comprising the following steps:

[0011] S1. Acquire multimodal magnetic resonance images and preprocess them;

[0012] The multimodal magnetic resonance images include anatomical magnetic resonance images, resting-state magnetic resonance images, and diffusion magnetic resonance images, and the magnetic resonance features obtained by preprocessing are anatomical features of structural magnetic resonance, temporal features of functional magnetic resonance, and structural connectivity matrix.

[0013] S2. Use graph embedding Node2vec to enhance the structural connectivity matrix to obtain a new structural connectivity matrix, which is used as input features together with anatomical features and temporal features;

[0014] S3, feature learning is performed on the input features through the branch attention module to obtain the attention structure connection features;

[0015] S4. The feature channel differentiation scores of the attention structure connection features and the input features are calculated through the graph feature constraint module, and then weighted learning is performed on different feature channels to obtain weighted structure connection features as the extracted brain network features.

[0016] Furthermore, step S2 includes the following sub-steps:

[0017] S21. From all structural connection matrices, select the top several brain regions with the strongest connections to each node in the structural connection, and then extract the corresponding subgraphs;

[0018] S22, perform walk sampling on the subgraph to obtain a walk sequence;

[0019] S23, taking the walk sequence as the input of the Skip-gram model and maximizing its objective function, thereby learning the node embedding vector in the hidden layer of the Skip-gram model;

[0020] S24. Calculate the cosine similarity based on the learned node embedding vector, and then learn a new structural connection matrix as the input feature.

[0021] Furthermore, in step S22, when walking and sampling on the subgraph, each node is used as the starting node, and the node to be sampled next is determined by calculation. The probability P of , and then determine the next sampling node;

[0022] Among them, the probability P is expressed as:

[0023]

[0024]

[0025]

[0026] Where, Indicates in i The nodes at which the steps are walked, Indicates in i- The node where the walk is in 1 step, Indicates the current node, Represents the target node, that is, the node to be sampled in the next step, represents the unnormalized transition probability, represents the normalization constant, represents the edges of the subgraph, From node t to node The weight coefficient of Indicates the transition from the current node v to the node The transition probability, Represents node t to node The shortest path distance, p represents the probability adjustment parameter for returning to the previous node during the wandering process, q represents the probability control parameter for visiting distant nodes, and node t is the previous node of the current node.

[0027] Furthermore, in step S23, the Skip-gram model aims to maximize the co-occurrence probability of the central node and its context nodes, and its objective function is expressed as:

[0028]

[0029] Where, Represents the one-hot encoding of the node, Indicates the number of nodes, represents the set of surrounding neighbors sampled according to the node and the set window size, Represents the probability of the nodes around the predicted node.

[0030] Furthermore, the branch attention module in step S3 includes a channel maximum pooling unit, a channel average pooling unit, a first MLP layer and a second MLP layer;

[0031] The step S3 includes the following sub-steps:

[0032] S31, performing channel maximum pooling processing and channel average pooling processing on the input features through the channel maximum pooling unit and the channel average pooling unit, respectively, to obtain a first feature and a second feature;

[0033] S32, performing cross-channel learning on the first feature and the second feature through the first MLP layer and the second MLP layer, respectively, to obtain a third feature and a fourth feature;

[0034] S33, summing the third feature and the fourth feature element by element, and converting the sum into an attention score using an activation function;

[0035] S34. Perform a spatial graph convolution operation on the input features, and apply the attention score to the output features of the spatial graph convolution operation by element-by-element multiplication to obtain the attention structure connection features.

[0036] Furthermore, the first MLP layer and the second MLP layer each include two fully connected layers and activation function operations connected in sequence.

[0037] Furthermore, in step S34, the attention structure connection feature Expressed as:

[0038]

[0039]

[0040] Where, represents the attention score, represents the output features of the spatial graph convolution operation, represents the graph convolutional network, A represents the input feature, represents the degree matrix, X represents the input features used in the spatial graph convolution operation, W is the weight matrix, BN is batch normalization, represents the ReLu activation function, Represents the Hadamard product.

[0041] Furthermore, the step S4 includes the following sub-steps:

[0042] S41. Calculate the output features of the spatial graph convolution operation Cosine similarity with input feature X;

[0043] S42. Calculate the output features of the spatial graph convolution operation based on the calculated cosine similarity The difference score with the input feature X on the feature channel;

[0044] S43, multiplying the differentiation score by the input feature X element by element, filtering the information of the feature channel, and obtaining the graph feature constraint output feature;

[0045] S44. Add the graph feature constraint output feature and the attention structure connection feature element by element to obtain the weighted structure connection feature as the extracted brain network feature.

[0046] Furthermore, the method further comprises:

[0047] The extracted brain network features are processed by global average pooling and fully connected layers in sequence, and then the classification results of the brain network features are obtained through activation function prediction.

[0048] Furthermore, for the predicted brain network feature classification results, the cross entropy loss function is used to determine the difference between them and the actual classification results.

[0049] The beneficial effects of the present invention are:

[0050] 1) It combines functional magnetic resonance imaging, structural magnetic resonance imaging, and diffusion tensor imaging data to fully realize the fusion of multimodal medical imaging data, effectively making up for the limitations of traditional brain connectome research on single modality data.

[0051] 2) Using graph embeddings like Node2vec, we enhanced structural connectivity, more effectively defining node edges and overcoming the instability of diffusion tensor data when constructing structural connectivity. Using a branched attention module and a graph feature constraint module, we learned significant brain regions and key features closely associated with brain disease.

[0052] 3) Compared with traditional statistical analysis methods, which are more difficult to identify patients at an individualized level, this approach can more comprehensively explore the topological structure of brain diseases by extracting brain network features, thereby achieving individualized research. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Flowchart of the multimodal magnetic resonance large-scale brain network feature extraction method provided by the present invention. DETAILED DESCRIPTION

[0054] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0055] The embodiment of the present invention provides a method for extracting large-scale brain network features from multimodal magnetic resonance imaging. Figure 1 As shown, the following steps are included:

[0056] S1. Acquire multimodal magnetic resonance images and preprocess them;

[0057] The multimodal magnetic resonance images include anatomical magnetic resonance images, resting-state magnetic resonance images, and diffusion magnetic resonance images, and the magnetic resonance features obtained by preprocessing are anatomical features of structural magnetic resonance, temporal features of functional magnetic resonance, and structural connectivity matrix.

[0058] S2. Use graph embedding Node2vec to enhance the structural connectivity matrix to obtain a new structural connectivity matrix, which is used as input features together with anatomical features and temporal features;

[0059] S3, feature learning is performed on the input features through the branch attention module to obtain the attention structure connection features;

[0060] S4. The feature channel differentiation scores of the attention structure connection features and the input features are calculated through the graph feature constraint module, and then weighted learning is performed on different feature channels to obtain weighted structure connection features as the extracted brain network features.

[0061] In step S1 of the embodiment of the present invention, anatomical magnetic resonance images and resting-state functional magnetic resonance images are acquired using magnetic resonance imaging acquisition equipment and processed to obtain anatomical features of structural magnetic resonance and temporal features of functional magnetic resonance to respectively characterize the functional and structural features of the brain regions. For diffusion magnetic resonance images, a fiber bundle imaging method is introduced to obtain the structural connection matrix of the brain.

[0062] In step S2 of this embodiment of the present invention, the structural connectivity of the structural connectivity matrix obtained by diffusion tensor imaging is enhanced using a graph embedding Node2vec method to compensate for the instability of diffusion tensor imaging in constructing long-range connections. Specifically, step S2 in this embodiment includes the following sub-steps:

[0063] S21. From all structural connection matrices, select the top several brain regions with the strongest connections to each node in the structural connection, and then extract the corresponding subgraphs;

[0064] For example, the structural connectivity matrix of each subject is used as input, and the top 10 brain regions with the strongest connections to each node in the structural connectivity are selected. The edge information between these brain regions is retained, and the weights of other edges are set to 0, thereby obtaining a subgraph with strong connections in each brain network, and then extracting the subgraph from the structural connectivity matrix (SC);

[0065] S22, perform walk sampling on the subgraph to obtain a walk sequence;

[0066] Specifically, when walking and sampling on a subgraph, each node is used as the starting node, and the node to be sampled next is determined by calculation. The probability P of , and then determine the next sampling node;

[0067] Among them, the probability P is expressed as:

[0068]

[0069]

[0070]

[0071] Where, Indicates in i The nodes at which the steps are walked, Indicates in i- The node where the walk is in 1 step, Indicates the current node, Represents the target node, that is, the node to be sampled in the next step, represents the unnormalized transition probability, represents the normalization constant, represents the edges of the subgraph, From node t to node The weight coefficient of Indicates the transition from the current node v to the node The transition probability, Represents node t to node The shortest path distance, p represents the probability adjustment parameter for returning to the previous node during the wandering process, q represents the probability control parameter for visiting distant nodes, and node t is the previous node of the current node.

[0072] S23, taking the walk sequence as the input of the Skip-gram model and maximizing its objective function, thereby learning the node embedding vector in the hidden layer of the Skip-gram model;

[0073] Specifically, the Skip-gram model aims to maximize the co-occurrence probability of the central node and its context nodes, and its objective function is expressed as:

[0074]

[0075] Where, Represents the one-hot encoding of the node, Indicates the number of nodes, represents the set of surrounding neighbors sampled according to the node and the set window size, Represents the probability of the nodes around the predicted node.

[0076] S24. Calculate the cosine similarity based on the learned node embedding vector, and then learn a new structural connection matrix as the input feature;

[0077] Specifically, the new structural connectivity matrix , That is, the adjacency matrix of the graph , which is expressed as:

[0078]

[0079] Where, , represents the learned node embedding.

[0080] In the embodiment of the present invention, a branch attention module is designed to integrate the channel attention mechanism into the graph data mechanism to achieve attention learning of significant brain areas. The branch attention module in this embodiment includes a channel maximum pooling unit, a channel average pooling unit, a first MLP layer and a second MLP layer. Based on this, step S3 in this embodiment includes the following sub-steps:

[0081] S31, performing channel maximum pooling processing and channel average pooling processing on the input features through the channel maximum pooling unit and the channel average pooling unit, respectively, to obtain a first feature and a second feature;

[0082] Specifically, the first feature and the second feature Respectively expressed as:

[0083]

[0084]

[0085] In the formula, MaxPool() represents channel maximum pooling, AvgPool() represents channel average pooling, and X represents input features;

[0086] S32, performing cross-channel learning on the first feature and the second feature through the first MLP layer and the second MLP layer, respectively, to obtain a third feature and a fourth feature;

[0087] Among them, the first MLP layer and the second MLP layer each include two fully connected layers and activation function operations connected in sequence.

[0088] Specifically, the third feature and the fourth characteristic Respectively expressed as:

[0089]

[0090]

[0091] Where, and They represent the weight matrix of the first fully connected layer respectively. and They represent the weight matrix of the second fully connected layer, and σ() represents the ReLU activation function;

[0092] S33, summing the third feature and the fourth feature element by element, and converting the sum into an attention score using an activation function;

[0093] Specifically, the attention score Expressed as:

[0094]

[0095] Where, Represents the sigmoid activation function;

[0096] S34. Perform a spatial graph convolution operation on the input features, and apply the attention score to the output features of the spatial graph convolution operation by element-by-element multiplication to obtain the attention structure connection feature;

[0097] Among them, the attention structure connection feature Expressed as:

[0098]

[0099]

[0100] Where, represents the attention score, represents the output features of the spatial graph convolution operation, represents the graph convolutional network, A represents the input feature, represents the degree matrix, X represents the input features used in the spatial graph convolution operation, W is the weight matrix, BN is batch normalization, represents the ReLu activation function, Represents the Hadamard product, which is the element-wise multiplication symbol.

[0101] In an embodiment of the present invention, a method for calculating feature channel differentiation scores is proposed. By calculating the feature channel differentiation scores of the features after the spatial graph convolution operation and the input features, and performing weighted learning on different feature channels based on the scores, important features that contribute to the model are obtained. This module not only helps to reduce the interference of redundant information, but also enhances the model's sensitivity to capturing key features, thereby improving the accuracy and robustness of the entire model.

[0102] Based on this, step S4 of the embodiment of the present invention includes the following sub-steps:

[0103] S41. Calculate the output features of the spatial graph convolution operation Cosine similarity with input feature X.

[0104] Specifically, the similarity between different channel features is measured by calculating the cosine similarity, which is expressed as:

[0105]

[0106] Where, 、 Respectively represent the feature vectors of each channel in the feature matrix of node i and node j after the spatial graph convolution operation;

[0107] S42. Calculate the output features of the spatial graph convolution operation based on the calculated cosine similarity The difference score with the input feature X on the feature channel.

[0108] Specifically, the differentiation score is expressed as:

[0109]

[0110] In the formula, Nor() represents the normalization operation, which constrains the cosine similarity result to the range of [0,1], and Score represents the output feature of the spatial graph convolution operation. The difference score between the input feature X;

[0111] S43. Multiply the differentiation score by the input feature X element by element, filter the information of the feature channel, and obtain the graph feature constraint output feature, which is expressed as:

[0112]

[0113] Where Z represents the output feature constrained by the graph feature.

[0114] S44. Add the graph feature constraint output feature and the attention structure connection feature element by element to obtain the weighted structure connection feature as the extracted brain network feature , which is expressed as:

[0115]

[0116] Where, Represents the element-wise addition symbol.

[0117] In an embodiment of the present invention, the brain network feature extraction method further includes:

[0118] The extracted brain network features are processed by global average pooling and fully connected layers in turn, and then the classification results of the brain network features are obtained by activation function prediction, which is expressed as:

[0119]

[0120] Where, represents the predicted brain network feature classification label, Represents the output after learning through two layers of branch attention and graph feature constraints, represents the fully connected layer, GavgPool() is the global average pooling operation, and σ() is the sigmoid activation function, which converts the output into a probability value for further classification decision-making.

[0121] Furthermore, for the predicted brain network feature classification results, the cross entropy loss function is used to determine the difference between them and the actual classification results, where the cross entropy loss function Expressed as:

[0122]

[0123] Where y represents the true label value of the brain network feature, Represents the predicted label value of the brain network feature sample.

[0124] Based on the above scheme, this embodiment uses medical MRI data, obtains the brain's structural connectivity matrix through a series of preprocessing operations, extracts structural and functional features, and integrates functional MRI, structural MRI, and diffusion tensor imaging information. Node2vec graph embedding is used to enhance structural connectivity, and branch attention and graph feature constraints are used to learn the characteristics of brain region nodes. This solution effectively solves the problem of multimodal information fusion for brain diseases, overcomes the shortcomings of using diffusion tensor imaging to construct structural connectivity matrices, and enables the learning of significant brain regions and important features related to brain diseases.

[0125] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

[0126] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A multimodal magnetic resonance large-scale brain network feature extraction method, characterized in that: The following steps are involved: S1. Acquire multimodal magnetic resonance images and preprocess them; The multimodal magnetic resonance images include anatomical magnetic resonance images, resting-state magnetic resonance images, and diffusion magnetic resonance images, and the magnetic resonance features obtained by preprocessing are anatomical features of structural magnetic resonance, temporal features of functional magnetic resonance, and structural connectivity matrix. S2. Use graph embedding Node2vec to enhance the structural connectivity matrix to obtain a new structural connectivity matrix, which is used as input features together with anatomical features and temporal features; S3, feature learning is performed on the input features through the branch attention module to obtain the attention structure connection features; S4. Calculate the feature channel differentiation scores of the attention structure connection features and the input features through the graph feature constraint module, and then perform weighted learning on different feature channels to obtain weighted structure connection features as the extracted brain network features; The branch attention module in step S3 includes a channel maximum pooling unit, a channel average pooling unit, a first MLP layer and a second MLP layer; The step S3 includes the following sub-steps: S31, performing channel maximum pooling processing and channel average pooling processing on the input features through the channel maximum pooling unit and the channel average pooling unit, respectively, to obtain a first feature and a second feature; S32, performing cross-channel learning on the first feature and the second feature through the first MLP layer and the second MLP layer, respectively, to obtain a third feature and a fourth feature; S33, summing the third feature and the fourth feature element by element, and converting the sum into an attention score using an activation function; S34. Perform a spatial graph convolution operation on the input features, and apply the attention score to the output features of the spatial graph convolution operation by element-by-element multiplication to obtain the attention structure connection feature; The step S4 comprises the following sub-steps: S41. Calculate the output features of the spatial graph convolution operation Cosine similarity with input feature X; S42. Calculate the output features of the spatial graph convolution operation based on the calculated cosine similarity The difference score with the input feature X on the feature channel; S43, multiplying the differentiation score by the input feature X element by element, filtering the information of the feature channel, and obtaining the graph feature constraint output feature; S44. Add the graph feature constraint output feature and the attention structure connection feature element by element to obtain the weighted structure connection feature as the extracted brain network feature.

2. The multimodal magnetic resonance large-scale brain network feature extraction method according to claim 1, characterized in that: The step S2 comprises the following sub-steps: S21. From all structural connection matrices, select the top several brain regions with the strongest connections to each node in the structural connection, and then extract the corresponding subgraphs; S22, perform walk sampling on the subgraph to obtain a walk sequence; S23, taking the walk sequence as the input of the Skip-gram model and maximizing its objective function, thereby learning the node embedding vector in the hidden layer of the Skip-gram model; S24. Calculate the cosine similarity based on the learned node embedding vector, and then learn a new structural connection matrix as the input feature.

3. The multimodal magnetic resonance large-scale brain network feature extraction method according to claim 2, characterized in that: In step S22, when walking and sampling on the subgraph, each node is used as the starting node, and the node to be sampled next is determined by calculation. The probability P of , and then determine the next sampling node; Among them, the probability P is expressed as: Where, Indicates in i The nodes at which the steps are walked, Indicates in i- The node where the walk is in 1 step, Indicates the current node, Represents the target node, that is, the node to be sampled in the next step, represents the unnormalized transition probability, represents the normalization constant, represents the edges of the subgraph, From node t to node The weight coefficient of Indicates the transition from the current node v to the node The transition probability, Represents node t to node The shortest path distance, p represents the probability adjustment parameter for returning to the previous node during the wandering process, q represents the probability control parameter for visiting distant nodes, and node t is the previous node of the current node.

4. The multimodal magnetic resonance large-scale brain network feature extraction method according to claim 2, characterized in that: In step S23, the Skip-gram model aims to maximize the co-occurrence probability of the central node and its context nodes, and its objective function is expressed as: Where, Represents the one-hot encoding of the node, Indicates the number of nodes, represents the set of surrounding neighbors sampled according to the node and the set window size, Represents the probability of the nodes around the predicted node.

5. The multimodal magnetic resonance large-scale brain network feature extraction method according to claim 1, characterized in that: The first MLP layer and the second MLP layer each include two fully connected layers and activation function operations connected in sequence.

6. The multimodal magnetic resonance large-scale brain network feature extraction method according to claim 1, characterized in that: In step S34, the attention structure connection feature Expressed as: Where, represents the attention score, represents the output features of the spatial graph convolution operation, represents the graph convolutional network, A represents the input feature, represents the degree matrix, X represents the input features used in the spatial graph convolution operation, W is the weight matrix, BN is batch normalization, represents the ReLu activation function, represents the Hadamard product.

7. The multimodal magnetic resonance large-scale brain network feature extraction method according to claim 1, characterized in that: The method further comprises: The extracted brain network features are processed by global average pooling and fully connected layers in sequence, and then the classification results of the brain network features are obtained through activation function prediction.

8. The multimodal magnetic resonance large-scale brain network feature extraction method according to claim 7, characterized in that: For the predicted brain network feature classification results, the cross entropy loss function is used to determine the difference between them and the actual classification results.

Images