Method for constructing a structure-function brain network bidirectional mapping model and a brain network bidirectional mapping model

Abstract

The present invention relates to a method for constructing a structure-function brain network bidirectional mapping and a brain network bidirectional mapping model, the method comprising: constructing a feature preprocessing module to obtain a brain structure network and a brain function network; constructing a structure feature extraction module and a function feature extraction module to obtain the structure feature and function feature of the brain; constructing a structure classifier module and a function classifier module to obtain a disease classification result based on the structure feature and a disease classification result based on the function feature; constructing a structure-function bidirectional mapping network to perform bidirectional mapping on the brain structure network and the brain function network; using the preprocessed data sets of the brain structure network and the brain function network, the constructed structure feature extraction module and the function feature extraction module, the structure classifier module and the function classifier module, and the structure-function bidirectional mapping network are trained and learned. The constructed brain network model is helpful to reveal the complex relationship between brain structure and function.

Description

Technical Field

[0001] The present invention relates to the field of brain network models, and in particular to a method for constructing a structure-function brain network bidirectional mapping model, a brain network bidirectional mapping model, and a computer-readable storage medium. Background Art

[0002] Alzheimer's disease is a common, irreversible dementia whose incidence increases significantly with age, making research on the disease urgent. With the continuous advancement of science and technology in recent years, our understanding of the brain has been continuously improving. Effective use of these imaging technologies can assist doctors in diagnosing and understanding the causes of the disease, providing a basis for subsequent treatment.

[0003] Currently, some researchers are using brain structural data to aid in the diagnosis of Alzheimer's disease, achieving promising results. Among current studies using graph structures to study brain diseases, some techniques utilize multiple graph convolutional models to fuse DTI features from different orientations, achieving intramodal fusion. Other techniques utilize not only structural and functional MRI data but also non-imaging data, improving the accuracy of MCI diagnosis and demonstrating the importance of prior knowledge in disease diagnosis. Other techniques utilize both DTI and fMRI data to construct supervised classification models. However, current approaches either focus solely on structure or function, ignoring the complementary relationship between structure and function, or simply consider a single directional relationship between structure and function, ignoring potentially complex relationships. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for constructing a structure-function brain network bidirectional mapping model, a brain network bidirectional mapping model and a computer-readable storage medium, which have the characteristics of making the constructed structure-function brain network bidirectional mapping model helpful to reveal the complex relationship between brain structure and function.

[0005] According to the present invention, a method for constructing a structure-function brain network bidirectional mapping model is provided, which specifically includes:

[0006] Constructing a feature preprocessing module, wherein the feature preprocessing module is used to preprocess the brain structural modality images and the brain functional modality images of patients of a specified type to obtain a brain structural network including corresponding node features and edge information, and to preprocess the brain functional modality images to obtain a brain functional network including corresponding node features and edge information;

[0007] Constructing a structural feature extraction module and a functional feature extraction module, wherein the structural feature extraction module is used to extract features of the brain structural network based on the cyclic graph convolution algorithm to obtain the structural features of the brain; the functional feature extraction module is used to extract features of the brain functional network based on the cyclic graph convolution algorithm to obtain the functional features of the brain;

[0008] Constructing a structural classifier module and a functional classifier module, wherein the structural classifier module is used to diagnose the patient's condition based on the structural features of the brain to obtain a condition classification result based on the structural features, thereby implementing training for the structural feature extraction module; and the functional classifier module is used to diagnose the patient's condition based on the functional features of the brain to obtain a condition classification result based on the functional features, thereby implementing training for the functional feature extraction module;

[0009] Constructing a structure-function bidirectional mapping network, and based on the structural characteristics and functional characteristics of the brain, using a structure generator module, a function generator module, a structure discriminator module, and a function discriminator module to perform bidirectional mapping between the brain structure network and the brain function network;

[0010] The preprocessed data sets of the brain structure network and brain function network are used to train and learn the constructed structural feature extraction module, functional feature extraction module, structural classifier module, functional classifier module and structure-function bidirectional mapping network.

[0011] By adopting the above technical solution, in a data-driven mode, the extraction of multimodal features and the collaborative training of brain network bidirectional mapping are realized, which is conducive to the constructed brain network bidirectional mapping model to successfully learn the potential feature representation of brain structure and function, and help reveal the relationship between brain structure and function while assisting in predicting disease categories.

[0012] Optionally, the specific method of constructing the structure-function bidirectional mapping network includes:

[0013] S101, based on the structural characteristics and functional characteristics of the brain, a structure generator generates a brain functional network G2' from the brain structural network G1, and a function generator generates a brain structural network G1' from the brain functional network G2;

[0014] S102, the function discriminator determines the difference between the generated brain function network G2' and the real brain function network G2, and the structure discriminator determines the difference between the brain structure network G1' and the real brain structure network G1;

[0015] S103, the function generator generates a brain structure network G1' from the brain function network G2', and the structure generator generates a brain function network G2' from the brain structure network G1';

[0016] S104, the function discriminator determines the difference between the generated brain function network G2" and the real brain function network G2, and the structure discriminator determines the difference between the generated brain structure network G1" and the real brain structure network G1.

[0017] Optionally, the steps between S101 and S102 include:

[0018] The brain structure data is fuzzified by performing multiple convolutions and activation function processing on the brain structure real connection matrix and the brain function real connection matrix, so that the brain structure real connection matrix contains part of the information of the brain function real connection matrix, thereby obtaining the fuzzified brain structure real connection matrix; this fuzzified brain structure real connection matrix and the brain structure generated connection matrix generated by the function generator are given to the structure discriminator;

[0019] The brain function data is fuzzified, and the brain function real connection matrix and the brain structure real connection matrix are subjected to multiple convolution and activation function processing, so that the brain function real connection matrix contains part of the information in the brain structure real connection matrix, thereby obtaining the fuzzified brain function real connection matrix; this fuzzified brain function real connection matrix and the brain function generated connection matrix generated by the structure generator are given to the function discriminator.

[0020] Optionally, the activation function adopts a ReLU activation function.

[0021] Optionally, the brain structure modality images include DTI and sMRI modality images; the brain function modality images include fMRI and PET modality images.

[0022] Optionally, the method for preprocessing the brain structural modal image includes: preprocessing the brain DTI modal image to obtain fiber bundles of brain white matter, mapping the fiber bundle image to a brain region division template, dividing the cerebral cortex and subcutaneous tissue into M brain regions, and completing the construction of the DTI brain structural network in combination with the M brain regions; preprocessing the brain sMRI modal image, mapping the sMRI brain modal image to the brain region division template to obtain the gray matter density of the M brain regions, and completing the construction of the sMRI brain structural network in combination with the gray matter density of the M brain regions, where the density of the fiber bundles between brain regions is the structural connection feature;

[0023] The method for preprocessing the brain functional modal images includes: preprocessing the brain fMRI modal images, mapping the fMRI modal image images to the brain region division template, obtaining the time series of M brain regions, and completing the construction of the fMRI brain functional network based on the time series of the M brain regions; preprocessing the brain PET, mapping the PET modal image images to the brain region division template, obtaining the content of labeled substances including protein, glucose, etc. in the M brain regions, and completing the construction of the PET brain network based on this.

[0024] Optionally, the brain region division template is an ALL template, and M=90.

[0025] Optionally, in the structural feature extraction module and the functional feature extraction module, based on the cyclic graph convolution formula

[0026]

[0027] Perform feature extraction; where * is the cyclic graph convolution operator, The operation form is shown below, ⊙ is the Hardmard product, is to perform some transformation on g. Similarly, is the same transformation of x, and U T (...) is to perform the inverse transformation;

[0028] Based on the cyclic graph convolution algorithm, the node information N1 and edge connection information E1 of the DTI brain structure network and the node information N2 and edge connection information E2 of the sMRI brain structure network are regarded as a group of brain structures, and the node information N3 and edge connection information E3 of the fMRI brain function network and the node information N4 and edge information E4 of the PET brain function network are regarded as a group of brain functions. Then, two feature extraction modules are used to extract the structural features (N1', N3') and functional features (N2', N4') of the brain respectively.

[0029] According to the present invention, a structure-function brain network bidirectional mapping model is provided, comprising:

[0030] a feature preprocessing module, based on brain structural modality images and brain functional modality images of patients of a specified type, preprocessing the brain structural modality images to obtain a brain structural network including corresponding node features and edge information, and preprocessing the brain functional modality images to obtain a brain functional network including corresponding node features and edge information;

[0031] The structural feature extraction module extracts features from the brain structure network based on the cyclic graph convolution algorithm to obtain the structural features of the brain;

[0032] Functional feature extraction module, based on the cyclic graph convolution algorithm, extracts features of the brain functional network to obtain the functional characteristics of the brain;

[0033] A structural classifier module diagnoses the patient's condition based on the structural features of the brain to obtain a condition classification result based on the structural features, so as to train the structural feature extraction module;

[0034] A functional classifier module diagnoses the patient's condition based on the functional characteristics of the brain to obtain a condition classification result based on the functional characteristics, so as to realize the training of the functional feature extraction module;

[0035] The structure-function bidirectional mapping network, based on the structural characteristics and functional characteristics of the brain, uses the structure generator module, function generator module, structure discriminator module and function discriminator module to perform bidirectional mapping on the brain structure network and the brain function network.

[0036] By adopting the above technical solutions, it is helpful to successfully learn the potential feature representation of brain structure and function, which can help reveal the relationship between brain structure and function while assisting in predicting disease categories.

[0037] Optionally, based on the structural characteristics and functional characteristics of the brain, first, the structure generator generates a brain function network G2' from the brain structure network G1, and the function generator generates a brain structure network G1' from the brain function network G2; secondly, the function discriminator judges the gap between the generated brain function network G2' and the real brain function network G2, and the structure discriminator judges the gap between the brain structure network G1' and the real brain structure network G1; thirdly, the function generator generates a brain structure network G1" from the brain function network G2', and the structure generator generates a brain function network G2" from the brain structure network G1'; finally, the function discriminator judges the gap between the generated brain function network G2" and the real brain function network G2, and the structure discriminator judges the gap between the generated brain structure network G1" and the real brain structure network G1.

[0038] Optionally, the structure-function bidirectional mapping network further includes a structure balancer and a function balancer;

[0039] The structural balancer includes a convolution layer and an activation function, which performs multiple convolution and activation function processes on the brain structure real connection matrix and the brain function real connection matrix, so that the brain structure real connection matrix contains part of the information in the brain function real connection matrix, thereby obtaining a fuzzified brain structure real connection matrix;

[0040] The functional balancer includes a convolution layer and an activation function, which performs multiple convolution and activation function processes on the brain function real connection matrix and the brain structure real connection matrix, so that the brain function real connection matrix contains part of the information in the brain structure real connection matrix, thereby obtaining a fuzzified brain function real connection matrix.

[0041] According to the present invention, a computer-readable storage medium is provided, which stores a computer program that can be loaded by a processor and execute any one of the above methods.

[0042] By adopting the above technical solution, it is conducive to the construction of the above brain network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of the principle of a structure-function bidirectional mapping brain network model according to one embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of a network structure for non-Euclidean spatial feature extraction according to one embodiment of the present invention;

[0045] Figure 3 It is a schematic diagram of the balancer structure of one embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below with reference to the accompanying drawings.

[0047] This specific embodiment is merely an explanation of the present invention and is not intended to limit the present invention. After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed. However, as long as such modifications are within the scope of the claims of the present invention, they are protected by patent law.

[0048] like Figure 1 As shown, the embodiment of the present invention provides a method for constructing a structure-function bidirectional mapping brain network model, and the specific method includes:

[0049] Constructing a feature preprocessing module, which is used to preprocess the brain structural modality images and brain functional modality images of patients of a specified type to obtain a brain structural network including corresponding node features and edge information, and to preprocess the brain functional modality images to obtain a brain functional network including corresponding node features and edge information;

[0050] Construct a structural feature extraction module and a functional feature extraction module. The structural feature extraction module is used to extract features of the brain structure network based on the cyclic graph convolution algorithm to obtain the structural features of the brain; the functional feature extraction module is used to extract features of the brain functional network based on the cyclic graph convolution algorithm to obtain the functional features of the brain;

[0051] Constructing a structural classifier module and a functional classifier module, wherein the structural classifier module is used to diagnose the patient's condition based on the structural features of the brain to obtain a condition classification result based on the structural features, thereby implementing training for the structural feature extraction module; and the functional classifier module is used to diagnose the patient's condition based on the functional features of the brain to obtain a condition classification result based on the functional features, thereby implementing training for the functional feature extraction module;

[0052] Construct a structure-function bidirectional mapping network. Based on the structural and functional characteristics of the brain, use the structure generator module, function generator module, structure discriminator module, and function discriminator module to perform bidirectional mapping on the brain structure network and the brain function network to generate a structure-function bidirectional mapping network.

[0053] Using the preprocessed data sets of the brain structure network and brain function network, the constructed structural feature extraction module, functional feature extraction module, structural classifier module, functional classifier module and structure-function bidirectional mapping network are trained and learned to obtain a structure-function bidirectional mapping brain network model based on multimodal data.

[0054] On the one hand, this application is based on the cyclic graph convolution algorithm, which not only considers the relationship within a single modality, but also the relationship between structure and function, so that all information within the modality can be fully utilized. On the other hand, although many articles at this stage use multimodal data, they often only discuss whether structure can generate function, or whether function can generate structure. In response to the above problems, the structure-function brain network bidirectional mapping model proposed in this application allows the model to simultaneously consider the mutual mapping relationship between structure and function.

[0055] Therefore, in a data-driven model, extracting multimodal features and co-training a bidirectional mapping network facilitates the construction of brain network models that successfully learn the underlying features of brain structure and function. This helps to predict disease categories while revealing the relationship between brain structure and function. Based on this, it can be determined which abnormal brain connections in brain structure or function are responsible for the disease, or whether abnormal brain activity is caused by misaligned structure and function.

[0056] Structure is the foundation of function, and function is a representation of structure. The relationship between structure and function plays a crucial role in revealing the organizational principles of the brain. This invention utilizes modal imaging data to construct brain structural and functional networks, and based on the relationships between these networks, assists in the diagnosis of brain diseases and elucidates their pathogenesis.

[0057] To illustrate the specific embodiments of the present invention, Alzheimer's disease (AD) is used as an example. Alzheimer's disease can be divided into five stages, which are categorized according to the order of disease progression: normal (NC), significant memory concern (SMC), early mild cognitive impairment (EMCI), late mild cognitive impairment (LMCI), and Alzheimer's disease (AD). In this example, N subjects were selected from each of the five stages, and brain structural and functional modality images were acquired for all 5*N subjects.

[0058] The multimodal recurrent graph convolutional network involved in the specific embodiments of this application considers four modalities for each subject sample: sMRI modality images, DTI modality images, fMRI modality images, and PET modality images. DTI and sMRI modality images are structural brain modality images, while fMRI and PET modality images are functional brain modality images. The four modal images of fMRI, DTI, sMRI, and PET are collected as raw data sets and then data conversion is performed on these raw data sets.

[0059] In the process of constructing the feature preprocessing module, the collected original data set is preprocessed to obtain the brain structure network and brain function network.

[0060] The method for preprocessing brain structural modal images includes: preprocessing brain DTI modal images to obtain fiber bundles of brain white matter, mapping the fiber bundle images to a brain region division template, dividing the cerebral cortex and subcutaneous tissue into M brain regions, and completing the construction of the DTI brain structural network by combining the M brain regions; preprocessing brain sMRI modal images, mapping the sMRI brain modal images to the brain region division template, obtaining the gray matter density of the M brain regions, and completing the construction of the sMRI brain structural network by combining the gray matter density of the M brain regions, where the density of the fiber bundles between brain regions is the structural connection feature;

[0061] The method for preprocessing brain functional modal images includes preprocessing brain fMRI modal images, mapping the fMRI modal image images to a brain region division template to obtain a time series of M brain regions, and completing the construction of the fMRI brain functional network based on the time series of the M brain regions; preprocessing brain PET, mapping the PET modal image images to a brain region division template to obtain the content of labeled substances including protein, glucose, etc. in the M brain regions, and completing the construction of the PET brain network based on this.

[0062] As one implementation method of preprocessing, PANDA software was used to preprocess DTI modal images, and the cerebral cortex and subcutaneous tissue were divided into 90 brain regions based on the AAL template; GRETNA software was used to preprocess fMRI modal images, and the fMRI images were mapped to the AAL template to obtain the time series of 90 brain regions; CAT12 software was used to preprocess sMRI modal images, and the sMRI modal images were mapped to the AAL template to obtain the brain gray matter density in 90 brain regions; NetPET software was used to preprocess PET modal images, and combined with the ALL template, the content of labeled substances including protein and glucose in M ​​brain regions was obtained. The scope of use of the preprocessing software of the present application is not limited to PANDA, GRETNA, CAT12 and NetPET software, as long as the preprocessing process can be implemented to obtain the corresponding brain network. It is not limited to the 90 brain regions of the ALL template, and other templates and other numbers of brain regions can also be used for division. For example, the 90 brain regions of the ALL1 template can be used, and the 116 brain regions of the ALL2 template can also be used. In certain specific cases, part of the 90 brain regions of the ALL1 template or part of the 116 brain regions of the ALL2 template can also be used.

[0063] After data conversion of the original dataset, the functional and structural brain networks were generated, including corresponding node and edge information. Specifically, PANDA software was used to extract node information N1 and edge connectivity information E1 from DTI modal images; Gretna software was used to extract time series from fMRI modal images, and Pearson correlation coefficients were used to construct node information N2 and edge connectivity information E2 from fMRI modal images; CAT12 software was used to extract gray matter density within the midbrain regions from sMRI images, and node information N3 and edge information E3 were constructed; and NetPET software was used to construct node information N4 and edge information E4 for the brain. Next, based on the recurrent graph convolution algorithm, node features were weighted and aggregated based on their connectivity relationships, and then dimensionality was reduced to a low-dimensional space, thereby extracting the features of the structural and functional brain networks separately, thereby obtaining the structural and functional characteristics of the brain.

[0064] In the process of extracting structural features and functional features, Figure 2 As shown in Figure 1, a non-Euclidean feature extraction network is used to transform the data of different modalities (such as Fourier transform and Z-domain transform). The data is mapped to the corresponding latent space (Fourier space, Z-domain space, etc.). Then, convolution is performed on the data. Due to the existence of the circular convolution mechanism, the features under different modalities are aggregated. The convolution result is then transformed back to the original space. The resulting features utilize the node and edge information of all modal data. The following is a detailed explanation of this process.

[0065] The brain network consists of two parts: the node feature matrix and the adjacency matrix.

[0066] It represents the adjacency matrix (i.e. edge matrix) of the graph; is a real number, n1 and n2 both represent the length of a dimension; a graph is a mathematical data structure, and a brain network refers to the name of the brain when it is represented as a graph data;

[0067] It represents the degree matrix of the graph. The degree matrix D has values ​​only on the diagonal, which are the degrees of the corresponding nodes, and the rest are 0;

[0068] It represents the Laplace matrix of the graph, which is calculated as Among them D (i) It represents D(:,:,i); I represents the unit tensor, and i represents a certain dimension;

[0069] It represents the convolution filter, and R represents the length of a certain dimension.

[0070] Represents the input feature matrix.

[0071] Using the formula

[0072]

[0073] Perform feature extraction; where * is the cyclic graph convolution operator, The operation form is shown below, ⊙ is the Hardmard product, is to perform some transformation on g. Similarly, is the same transformation of x, and U T (...) is the inverse transform. Among them, the calculation form of U (Fourier transform) is:

[0074] L dft =DFT(L)

[0075] for i in 1,2,3 do

[0076] [u,ξ]=EVD(L dft (i) )

[0077]

[0078] end for

[0079]

[0080] The operation form is:

[0081] Here we assume A∈R n1×n2×3 and F∈R n2×n4×3 ,and in U is the corresponding transformation form (Fourier transform, Z transform, etc.), and U H is the inverse transform.

[0082] Based on the cyclic graph convolution algorithm, N1, E1 and N3, E3 are regarded as a group of brain structures, and N2, E2 and N4, E4 are regarded as a group of brain functions. Then, two feature extraction modules (structural feature extraction module and functional feature extraction module) are used to extract the structural features and functional features (N1', N3') and functional features (N2', N4') of the brain respectively.

[0083] Currently, some researchers are using brain structural data to assist in the diagnosis of Alzheimer's disease, achieving promising results. Compared to healthy individuals (of the same age), patients exhibit significant atrophy in some brain regions, which impedes the performance of certain functions. Some researchers have measured levels of beta-amyloid protein in patients' brains, which are generally higher than those in healthy individuals. Other researchers are using brain functional data to study Alzheimer's disease, assessing the patient's condition based on resting brain activity over a specific time period. Most of the aforementioned methods are based on convolutional neural networks (CNNs). While CNNs are effective at extracting features, they spatially aggregate N*N adjacent nodes and temporally aggregate information from nodes within a specific time period. However, they overlook the fact that adjacent nodes may influence each other, but not necessarily the most significant influence. Instead, pixels farther apart have a greater degree of mutual influence.

[0084] To this end, more and more scholars are applying graph convolution to brain disease research. A common approach is to use one modality's data as the graph's edge connection information, and the other modality's data as the graph's node information. Compared to traditional graph convolution, the circular graph convolution used in the present invention expands its dimension, extending the original matrix calculation form to multi-dimensional tensor calculation. The benefit is that it can more fully utilize information from different modalities. In addition, due to the mechanism of circular convolution, it is also possible to try to use the relationship between modalities, thus obtaining richer information.

[0085] In disease classification, in order to fully utilize the feature information of non-Euclidean space, while reducing the computational complexity of the model and improving the efficiency of network collaborative training, we chose a simple and effective classifier in this application. It mainly includes a layer of recurrent graph convolution module and MLP (multi-layer perceptron) module. First, the node information and edge connection information are given to the recurrent graph convolution module, and then the further extracted information (90*1) is given to the MLP module. The MLP can comprehensively use the feature information of these 90 brain regions to give the final classification result. The feature extraction process is as follows:

[0086] The 90*d (d is the length of a dimension of the feature matrix; the length is 90, representing the number of brain regions; the width is d, representing the length of the feature vector for each brain region) four-modal brain network data ((N1', E1), (N3', E3), (N2', E2), and (N4', E4)) obtained during feature extraction are fused using a layer of recurrent graph convolution. The four 90*d feature matrices are fused into a single 90*1 feature matrix M3'. M3' is then fed into a fully connected layer, which weights the features of the 90 brain regions. After a softmax layer, the probabilities of the outputs for the five categories (Alzheimer's disease is divided into the following five categories based on the order of disease progression) are calculated.

[0087] In order to explore the relationship between structure and function, the present invention uses a generative adversarial network with bidirectional mapping. Based on the structural and functional characteristics of the brain, a structure generator module, a function generator module, a structure discriminator module, and a function discriminator module are used to perform bidirectional mapping between the brain structure network and the brain function network. The specific method for constructing a structure-function bidirectional mapping network includes:

[0088] S101, based on the structural characteristics and functional characteristics of the brain, the structure generator generates a brain functional network G2' from the brain structural network G1, and the function generator generates a brain structural network G1' from the brain functional network G2;

[0089] S102, the function discriminator determines the difference between the generated brain function network G2' and the real brain function network G2, and the structure discriminator determines the difference between the brain structure network G1' and the real brain structure network G1;

[0090] S103, the function generator generates a brain structural network G1' from the brain functional network G2', and the structure generator generates a brain functional network G2' from the brain structural network G1';

[0091] S104. The functional discriminator determines the difference between the generated brain functional network G2" and the real brain functional network G2, and the structural discriminator determines the difference between the generated brain structural network G1" and the real brain structural network G1.

[0092] from Figure 1 As can be seen from the figure, compared to the complex generator, the discriminator has a simple structure and function. This results in a difficult balance between the performance of the generator and the discriminator. Based on this, a balancer is added before the discriminator in S102 to slow down the learning rate of the discriminator during the generative adversarial process.

[0093] like Figure 3 The structure of the structure balancer shown in FIG2 includes a convolution layer and an activation function. Based on the structure of the structure balancer, according to the performance difference between the structure generator and the structure discriminator in the previous loop iteration process, the brain structure true connection matrix (i.e., the side information of the brain structure network) and the brain function true connection matrix (i.e., the side information of the brain function network) are subjected to multiple convolution and activation function processing, so that the brain structure true connection matrix contains part of the information in the brain function true connection matrix, thereby obtaining a fuzzified brain structure true connection matrix; this fuzzified brain structure true connection matrix and the brain structure generated connection matrix generated by the function generator are given to the structure discriminator, thereby reducing the gap between the generated result and the true result, thereby reducing the learning difficulty of the structure generator; at the same time, reducing the gap between the generated result and the true result can also slow down the rate at which the structure classifier fully learns the distribution of the true result data, thereby making the performance of the structure generator and the structure discriminator tend to be balanced.

[0094] The structure of the function balancer is identical to that of the structure balancer, including convolutional layers and activation functions. Based on the structure of the function balancer, and according to the performance difference between the function generator and the function discriminator during the previous iteration, the brain function true connection matrix and the brain structure true connection matrix are subjected to multiple convolutions and activation function processing. This allows the brain function true connection matrix to contain some information from the brain structure true connection matrix, thereby obtaining a fuzzified brain function true connection matrix. This fuzzified brain function true connection matrix and the brain function generated connection matrix generated by the structure generator are then fed to the function discriminator, thereby reducing the gap between the generated results and the true results, thereby reducing the learning difficulty of the function generator. At the same time, reducing the gap between the generated results and the true results can also slow the rate at which the function classifier fully learns the distribution of the true result data, thereby balancing the performance of the function generator and the function discriminator.

[0095] As a specific embodiment of the present invention, the convolutional layer has a convolution kernel of 3*3 and a stride of 1. The activation function can use the ReLU activation function, and four sets of convolutional layers and activation functions are performed. Parameter adjustment requires experimental trial and error. The remaining parameters can be controlled unchanged, and then the current parameters can be modified. After multiple experiments, the optimal number of convolutional layers and activation functions can be found within these parameters.

[0096] Based on the constructed model framework, the model is trained. During the training process, data from four modalities (fMRI, DTI, sMRI, and PET) are input. sMRI and DTI are used as brain structural networks to extract the structural feature representation of the brain, and fMRI and PET are used as brain functional networks to extract the functional feature representation of the brain. The two feature representations are given one-to-one correspondence to the structural classifier module and the functional classifier module respectively, and the disease prediction category is output.

[0097] Based on the classification output results of the structure classifier module (or function classifier module), the loss function of the structure classifier module (or function classifier module) is obtained by comparing it with the actual classification results.

[0098]

[0099] Here, the main method used is multi-classification experiment. K is the number of categories; y i is the label of category i, y i =1 or y i =0;p i is the output of the structure classifier module (or function classifier module), that is, the probability of category i. This output value is calculated using softmax. The loss function of the structure classifier module (or function classifier module) is mainly used for backpropagation learning training to evaluate the structure feature extraction module (or function feature extraction module).

[0100] Since the loss function of the structural classifier and the loss function algorithm of the functional classifier are the same, according to the loss function algorithm of the structural classifier given above, the loss function algorithm of the functional classifier after replacing the functional classifier in the parentheses with the structural classifier can be obtained. The following parentheses represent the same meaning.

[0101] In order to slow down the learning rate of the structure classifier module (or function classifier module), the performance difference between the structure generator module (or function generator module) and the structure discriminator module (or function discriminator module) is considered, and the loss function of the structure balancer (or function balancer) is obtained.

[0102]

[0103] Among them, X is the input of the model, Y is the generated result of the model, D is the abbreviation of the structure discriminator module (or function discriminator module), and α is a hyperparameter that can be adjusted according to the performance difference between the structure generator module (or function generator module) and the structure discriminator module (or function discriminator module).

[0104]

[0105] For the loss function of the structure-function bidirectional mapping network, its calculation formula is

[0106]

[0107] It mainly includes the following parts: (1) the loss function of the structure generator module (or function generator module) in the bidirectional mapping network plus the loss function of the structure discriminator module (or function discriminator module) in the bidirectional mapping network This loss function is mainly used for back-propagation learning training to evaluate the structural feature extraction module (or functional feature extraction module), the structure generator module (or functional generator module), and the structure discriminator module (or functional discriminator module). (2) The difference between the results generated by the structure generator module (or functional generator module) and the true results in the bidirectional mapping network This loss function is mainly used for back propagation learning training to evaluate the structural feature extraction module, the structure generator module, and the function generator module. (3) The difference between the generator-generated result G2" and the true result G2 in the bidirectional mapping network This loss function is mainly used for back propagation learning training of the functional feature extraction module, structure generator module and function generator module evaluation. (4) In the bidirectional mapping network, the structure generator not only learns the mapping relationship from G1 to G2', but also needs to learn the mapping relationship from G1 to G1' during each training process. This loss function is mainly used for the directional propagation learning training to evaluate the structural feature extraction module and the structure generator module. (5) Classification results of the structure classifier module (or function classifier module) in the bidirectional mapping network It will be used as one of the evaluation criteria for the performance of the structure generator module (or function generator module); the loss function is mainly used for the directional propagation learning training for evaluating the structure (or function) feature extraction module and the structure (or function) classifier.

[0108] The training samples were divided into 90% for model training and 10% for testing the performance of the trained model. The model training process can be qualitatively analyzed as follows: Under a data-driven model, as the bidirectional mapping network and recurrent graph convolution are continuously optimized, the structural discriminator module and the functional discriminator module continuously update their parameters to identify the pseudo-data distribution generated by the structural generator module and the functional generator module. The structural generator module and the functional generator module continuously update their parameters based on the results of the structural discriminator module and the functional discriminator module, thereby bringing the generated distribution closer to the true distribution. The structural classifier module and the functional classifier module are optimized to ensure that their predicted data distribution has good discrimination between different categories. After model training is complete, the structural classifier module and the functional classifier module can assist doctors in auxiliary diagnosis. Furthermore, the bidirectional mapping network can achieve mutual mapping between structural and functional brain networks. Based on this mechanism, it can be used to explore whether there is a unified relationship between structure and function in brain diseases.

[0109] Based on the application scheme of the present invention, the mutual mapping between structure and function is carried out, and the effectiveness of the generated results is tested, which proves that the mapping relationship learned by the structure generator module and the function generator module is meaningful.

[0110] The validity test steps are as follows:

[0111] 1) Using the trained structure generator module and function generator module to verify the effectiveness of the generated results, the classification results obtained were similar or even better, indicating that the structure generator module and function generator module learned the mapping relationship between structure and function and even optimized the original brain network.

[0112] 2) Misaligned learning: When combining the structural and functional brain networks of different patients, the structures and functions cannot be mapped to each other, or the classification effect is poor, indicating that neither the classifier nor the generator module can learn the relationship between structure and function.

[0113] To investigate the relationship between structure and function, a statistical approach was used to investigate the relationship between structure and function in patients. The steps were as follows:

[0114] The number of structural and functional brain connections in different patient categories was counted. There were significant differences in the number of structural and functional brain connections between different patient categories, and the trends in these numbers also showed significant differences.

[0115] Comparing the brain structure and functional brain networks of a single individual revealed that only a few brain regions shared significant similarities between the structural and functional networks of the individuals, demonstrating that there is not a simple one-to-one correspondence between brain structure and functional networks.

[0116] Statistical methods were used to investigate structural and functional trends across patient categories. The distribution of brain structure and functional networks varied across patient categories. The abnormal structural and functional brain connections identified using this method were only partially consistent, while some connections exhibited significant structural and functional differences. This demonstrates the complex relationship between structure and function, rather than a simple one-to-one correspondence; a single function can require the collaboration of multiple brain regions.

[0117] According to the present invention, a structure-function brain network bidirectional mapping model is provided, comprising:

[0118] A feature preprocessing module, based on the brain structural modality images and brain functional modality images of patients of a specified type, preprocesses the brain structural modality images to obtain a brain structural network including corresponding node features and edge information, and preprocesses the brain functional modality images to obtain a brain functional network including corresponding node features and edge information;

[0119] The structural feature extraction module extracts features from the brain structure network based on the cyclic graph convolution algorithm to obtain the structural features of the brain;

[0120] Functional feature extraction module, based on the cyclic graph convolution algorithm, extracts features of the brain functional network to obtain the functional characteristics of the brain;

[0121] A structural classifier module diagnoses the patient's condition based on the structural features of the brain to obtain a condition classification result based on the structural features, so as to train the structural feature extraction module;

[0122] A functional classifier module diagnoses the patient's condition based on the functional characteristics of the brain to obtain a condition classification result based on the functional characteristics, so as to realize the training of the functional feature extraction module;

[0123] The structure-function bidirectional mapping network, based on the structural characteristics and functional characteristics of the brain, uses the structure generator module, function generator module, structure discriminator module and function discriminator module to perform bidirectional mapping on the brain structure network and the brain function network.

[0124] Based on the bidirectional structural-functional brain network mapping model constructed in this application, corresponding abnormal brain connections were identified. Structural and functional classifiers were constructed based on these abnormal brain connections. If the final classification results were similar to those above, it could be concluded that these brain connections indeed played a significant role in the disease. Individual brain connections were deleted one by one, ensuring that the remaining variables remained the same, and the experiment was repeated to confirm the relationship between individual brain connections and the disease.

[0125] Based on the above brain network model, a multimodal image fusion-assisted diagnosis system can be provided. Through iterative training of the network, the network reaches its optimal state in adversarial training. At this time, the first recurrent graph convolution module will learn the features mapped from sMRI and DTI to the latent space, while the second recurrent graph convolution module will learn the features mapped from fMRI to the latent space. The classifier learns the optimal parameters for multimodal fusion and can relatively accurately predict the corresponding disease category and corresponding brain connectivity model from the four input modal data. The trained generator module and classifier module are extracted and transferred to form an end-to-end multimodal image fusion-assisted diagnosis system.

[0126] When the auxiliary diagnosis system is used, it collects data from four imaging modalities: sMRI, fMRI, DTI, and PET. This multimodal data is input into a trained multimodal fusion network to complete the classification diagnosis of patients' cognitive diseases. The generator module effectively learns the relationship between structure and function, mapping them and optimizing the original brain atlas. This not only improves classification accuracy but also helps researchers explore the underlying coordination mechanisms between brain structure and function.

[0127] In addition, it should be noted that Alzheimer's disease is only one type of patient selected in the embodiment of the present invention. The four modal images of DTI, fMRI, sMRI, and PET are used as examples for illustration. However, the application scope of this application is not limited to Alzheimer's disease and fMRI-DTI-sMRI-PET images. It can also be other types of patients. As a type of patient, 5 stages are selected here. During the experimental verification process, several stages can also be selected or not limited to the 5 stages. For the acquired structural brain modal images, they are not limited to DTI and sMRI modal images. They can be one or several modal images, or other modal images, as long as they are structural brain modal images. For the acquired functional brain modal images, they are not limited to fMRI and PET modal images. They can be one or several modal images, or other modal images, as long as they are functional brain modal images.

[0128] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0129] The embodiments of this specific implementation method are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing a bidirectional mapping model of structure-function brain network, characterized in that: Specific methods include: Constructing a feature preprocessing module, wherein the feature preprocessing module is used to preprocess the brain structure modality images and the brain function modality images of patients of a specified type to obtain a brain structure network including corresponding node features and edge information, and to preprocess the brain function modality images to obtain a brain function network including corresponding node features and edge information; Constructing a structural feature extraction module and a functional feature extraction module, wherein the structural feature extraction module is used to extract features of the brain structure network based on a cyclic graph convolution algorithm to obtain the structural features of the brain; the functional feature extraction module is used to extract features of the brain function network based on a cyclic graph convolution algorithm to obtain the functional features of the brain; Constructing a structural classifier module and a functional classifier module, wherein the structural classifier module is used to diagnose the patient's condition based on the structural features of the brain to obtain a condition classification result based on the structural features, so as to realize the training of the structural feature extraction module; The functional classifier diagnoses the patient's condition based on the functional characteristics of the brain to obtain a condition classification result based on the functional characteristics, so as to realize the training of the functional characteristic extraction module; Constructing a structure-function bidirectional mapping network, based on the structural characteristics and functional characteristics of the brain, using a structure generator module, a function generator module, a structure discriminator module and a function discriminator module to perform bidirectional mapping between the brain structure network and the brain function network; Using the preprocessed data sets of the brain structure network and brain function network, training and learning the constructed structural feature extraction module, functional feature extraction module, structural classifier module, functional classifier module and structure-function bidirectional mapping network; The brain structure modality images include DTI and sMRI modality images; the brain function modality images include fMRI and PET modality images; The step of preprocessing the brain structure modality image to obtain a brain structure network including corresponding node features and edge information includes: Preprocessing the brain DTI modality images to obtain fiber bundle images of brain white matter, and mapping the fiber bundle images to the brain region division template, dividing the cerebral cortex and subcutaneous tissue into M brain regions, and completing the construction of the DTI brain structure network in combination with the M brain regions; Preprocess the brain sMRI modality images, map the sMRI brain modality images to the brain region division template, obtain the gray matter density of M brain regions, and complete the construction of the sMRI brain structure network in combination with the gray matter density of M brain regions; The preprocessing of the brain function modality image to obtain a brain function network including corresponding node features and edge information includes: Preprocess the fMRI modality images of the brain, map the fMRI modality images to the brain region division template, obtain the time series of M brain regions, and complete the construction of the fMRI brain functional network by combining the time series of the M brain regions; Preprocess the brain PET, map the PET modality image to the brain region division template, and obtain the content of labeled substances including protein and glucose in M ​​brain regions; In the structural feature extraction module and the functional feature extraction module, based on the cyclic graph convolution formula Perform feature extraction; where * is the cyclic graph convolution operator, ⊙ is the Hardmard product, is to perform some transformation on g. Similarly, is the same transformation of x, and U T (...) is correct Perform the inverse transformation, The operation form is as follows: Let A∈R n1×n2×3 and F∈R n2×n4×3 , in, U is the corresponding transformation form, U H is the inverse transform; Based on the cyclic graph convolution algorithm, the node information N1 and edge connection information E1 of the DTI brain structure network and the node information N3 and edge connection information E3 of the sMRI brain structure network are taken as a group of brain structures, and the node information N2 and edge connection information E2 of the fMRI brain function network and the node information N4 and edge information E4 of the PET brain function network are taken as a group of brain functions. Then, two feature extraction modules are used to extract the structural features (N1', N3') and functional features (N2', N4') of the brain one by one.

2. The method for constructing a structure-function brain network bidirectional mapping model according to claim 1, characterized in that: The specific method of constructing the structure-function bidirectional mapping network includes: S101, based on the structural characteristics and functional characteristics of the brain, the structure generator generates a brain functional network G2′ from the brain structural network G1, and the function generator generates a brain structural network G1′ from the brain functional network G2; S102, the function discriminator determines the difference between the generated brain function network G2′ and the real brain function network G2, and the structure discriminator determines the difference between the brain structure network G1′ and the real brain structure network G1; S103, the function generator generates a brain structure network G1″ from the brain function network G2′, and the structure generator generates a brain function network G2″ from the brain structure network G1′; S104, the function discriminator determines the difference between the generated brain function network G2" and the real brain function network G2, and the structure discriminator determines the difference between the generated brain structure network G1" and the real brain structure network G1.

3. The method for constructing a structure-function brain network bidirectional mapping model according to claim 2, characterized in that: The S101 and S102 include: The brain structure data is fuzzified by performing multiple convolutions and activation function processing on the brain structure real connection matrix and the brain function real connection matrix, so that the brain structure real connection matrix contains part of the information in the brain function real connection matrix, thereby obtaining the fuzzified brain structure real connection matrix; The fuzzified real brain structure connection matrix and the brain structure generated connection matrix generated by the function generator are given to the structure discriminator; Fuzzy processing of brain function data, performing multiple convolution and activation function processing on the brain function real connection matrix and the brain structure real connection matrix, so that the brain function real connection matrix contains part of the information in the brain structure real connection matrix, thereby obtaining the fuzzy brain function real connection matrix; The fuzzified brain function true connection matrix and the brain function generated connection matrix generated by the structure generator are given to the function discriminator.

4. The method for constructing a structure-function brain network bidirectional mapping model according to claim 3, characterized in that: The activation function adopts the ReLU activation function.

5. A structure-function brain network bidirectional mapping model, characterized in that: include, A feature preprocessing module, based on the brain structure modality images and brain function modality images of patients of a specified type, preprocesses the brain structure modality images to obtain a brain structure network including corresponding node features and edge information, and preprocesses the brain function modality images to obtain a brain function network including corresponding node features and edge information; The structural feature extraction module extracts features from the brain structure network based on the cyclic graph convolution algorithm to obtain the structural features of the brain; Functional feature extraction module, based on the cyclic graph convolution algorithm, extracts features of the brain functional network to obtain the functional features of the brain; A structural classifier module diagnoses the patient's condition based on the structural features of the brain to obtain a condition classification result based on the structural features, so as to realize the training of the structural feature extraction module; A functional classifier module diagnoses the patient's condition based on the functional characteristics of the brain to obtain a condition classification result based on the functional characteristics, so as to realize the training of the functional characteristic extraction module; A structure-function bidirectional mapping network, based on the structural characteristics and functional characteristics of the brain, uses a structure generator module, a function generator module, a structure discriminator module and a function discriminator module to perform bidirectional mapping on the brain structure network and the brain function network; The brain structure modality images include DTI and sMRI modality images; the brain function modality images include fMRI and PET modality images; The feature preprocessing module preprocesses the brain structure modality image to obtain a brain structure network including corresponding node features and edge information, including: Preprocessing the brain DTI modality images to obtain fiber bundle images of brain white matter, and mapping the fiber bundle images to the brain region division template, dividing the cerebral cortex and subcutaneous tissue into M brain regions, and completing the construction of the DTI brain structure network in combination with the M brain regions; Preprocess the brain sMRI modality images, map the sMRI brain modality images to the brain region division template, obtain the gray matter density of M brain regions, and complete the construction of the sMRI brain structure network in combination with the gray matter density of M brain regions; The feature preprocessing module preprocesses the brain function modality image to obtain a brain function network including corresponding node features and edge information, including: Preprocess the fMRI modality images of the brain, map the fMRI modality images to the brain region division template, obtain the time series of M brain regions, and complete the construction of the fMRI brain functional network by combining the time series of the M brain regions; Preprocess the brain PET, map the PET modality image to the brain region division template, and obtain the content of labeled substances including protein and glucose in M ​​brain regions; In the structural feature extraction module and the functional feature extraction module, based on the cyclic graph convolution formula Perform feature extraction; where * is the cyclic graph convolution operator, ⊙ is the Hardmard product, is to perform some transformation on g. Similarly, is the same transformation of x, and U T (...) is correct Perform the inverse transformation, The operation form is as follows: Let A∈R n1×n2×3 and F∈R n2×n4×3 , in, U is the corresponding transformation form, U H is the inverse transform; Based on the cyclic graph convolution algorithm, the node information N1 and edge connection information E1 of the DTI brain structure network and the node information N3 and edge connection information E3 of the sMRI brain structure network are taken as a group of brain structures, and the node information N2 and edge connection information E2 of the fMRI brain function network and the node information N4 and edge information E4 of the PET brain function network are taken as a group of brain functions. Then, two feature extraction modules are used to extract the structural features (N1', N3') and functional features (N2', N4') of the brain one by one.

6. The structure-function brain network bidirectional mapping model according to claim 5, characterized in that: Based on the structural and functional characteristics of the brain, first, the structure generator generates a brain function network G2′ from the brain structure network G1, and the function generator generates a brain structure network G1′ from the brain function network G2; secondly, the function discriminator judges the difference between the generated brain function network G2′ and the real brain function network G2, and the structure discriminator judges the difference between the brain structure network G1′ and the real brain structure network G1; thirdly, the function generator generates a brain structure network G1″ from the brain function network G2′, and the structure generator generates a brain function network G2″ from the brain structure network 61′; finally, the function discriminator judges the difference between the generated brain function network G2″ and the real brain function network G2, and the structure discriminator judges the difference between the generated brain structure network G1″ and the real brain structure network G1.

7. The structure-function brain network bidirectional mapping module according to claim 5, characterized in that: The structure-function bidirectional mapping network also includes a structure balancer and a function balancer; The structural balancer includes a convolution layer and an activation function, which converts the real connection matrix of brain structure and the real connection matrix of brain function into The matrix is ​​processed with multiple convolutions and activation functions, so that the brain structure real connection matrix contains part of the information in the brain function real connection matrix, thereby obtaining the fuzzy brain structure real connection matrix; The functional balancer includes a convolution layer and an activation function, which performs multiple convolution and activation function processes on the brain function real connection matrix and the brain structure real connection matrix, so that the brain function real connection matrix contains part of the information in the brain structure real connection matrix, thereby obtaining a fuzzy brain function real connection matrix.

8. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 4.

Images