Multimodal Brain Network Computing Method, Device, Equipment and Medium Related to Structural and Functional Association

Abstract

The present invention discloses a multimodal brain network calculation method, device, equipment and medium with structural-functional association, which is applied to training a brain disease prediction model. The model includes an association perception dual-channel generation module, a disease feature regression module, a topological structure discriminator and a time-space joint discriminator. During the model training process, through multi-level interactive fusion learning of the high-order topological features of brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data, the multimodal time series activity signals of each brain region are obtained. Combining the directional holistic causal inference based on the structural equation of brain region activity, the directional causal relationship between each brain region is characterized, and the multimodal effective connectivity calculation is realized, so that the trained model can use the multimodal effective connectivity to perform intelligent assisted pathological analysis and lesion tracing for patients with neurodegenerative diseases.

Description

Technical Field

[0001] The present application relates to the technical field of machine learning, and particularly to a multimodal brain network calculation method, device, equipment and medium for structural-functional association. Background Art

[0002] Currently, brain diseases have become a common health problem in the world today, seriously endangering the lives of patients. Therefore, the detection and diagnosis of brain diseases have received more and more research attention. The research direction of brain connectivity is one of them. By analyzing brain connectivity, it helps in the diagnosis and pathological tracing of neurodegenerative diseases. Taking Alzheimer's disease (AD) as an example, patients with Alzheimer's disease will experience changes in brain connectivity during the disease development process. These change characteristics can be obtained through brain images such as fMRI and DTI. The traditional conventional method is that professional physicians set specific parameters through software templates, manually register, and correct images to obtain effective connections. This traditional method for analyzing pathological characteristics highly depends on the experience of professional physicians, has high time costs and labor costs, and the output effect is greatly affected by the parameter settings of the software template, which is not conducive to personalized and accurate diagnosis and treatment.

[0003] With the development of artificial intelligence technology, many brain connectivity intelligent calculation systems that do not rely on professional physicians have emerged. The effective connectivity intelligent calculation systems can be divided into two major categories: 1) effective connectivity intelligent calculation based on single modality; 2) effective connectivity intelligent calculation based on multimodality. However, the above-mentioned single modality signals mainly reflect the activity characteristics of brain regions, and the main defect is the lack of the nerve fiber structure characteristics between brain regions, resulting in the inability to use the overall brain topological structure information to guide the directional causal relationship between brain regions, thus limiting the learning ability and accuracy of the model. On the other hand, existing intelligent calculation systems based on multimodal neuroimaging data only use methods such as affine stitching or weighted summation to fuse multimodal data. The problem with these methods is that they ignore the heterogeneous-heterostructural nature of different modality data, so it is difficult to deeply mine the complementary information between different modalities, limiting the performance of the model and resulting in poor practicality and low accuracy of the finally obtained model. Summary of the Invention

[0004] In view of this, the present application provides a multimodal brain network calculation method, device, equipment and medium for structural-functional association to solve the problems of poor practicality and low accuracy of existing brain disease prediction models.

[0005] To solve the above technical problems, a technical solution adopted in this application is to provide a multimodal brain network calculation method with associated structure and function, which is applied to train a brain disease prediction model. The brain disease prediction model includes an association perception dual-channel generation module, a disease feature regression module, a topological structure discriminator, and a time-space joint discriminator. The method includes: obtaining brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data; inputting the brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data into the association perception dual-channel generation module for interactive association perception fusion to obtain multimodal brain region activity signal features, multimodal effective connection matrices, and reconstructed structural connection matrices; inputting the multimodal effective connection matrices into the disease feature regression module for prediction, inputting the reconstructed structural connection matrices into the topological structure discriminator for prediction, and inputting the multimodal brain region activity signal features into the time-space joint discriminator for prediction; and reversely updating the association perception dual-channel generation module, the disease feature regression module, the topological structure discriminator, and the time-space joint discriminator according to the prediction results and a pre-constructed loss function.

[0006] As a further improvement of this application, the association perception dual-channel generation module includes a brain region feature extraction module, a structure-to-function conversion module, a function-to-structure conversion module, an orientation overall causal inference module, and a structure decoding module.

[0007] As a further improvement of this application, inputting the brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data into the association perception dual-channel generation module for interactive association perception fusion to obtain multimodal brain region activity signal features, multimodal effective connection matrices, and reconstructed structural connection matrices includes: using the brain region feature extraction module to respectively extract first initial features and second initial features from the brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data; inputting the first initial features and the second initial features into the structure-to-function conversion module, and performing weighted fusion on the features output by the structure-to-function conversion module and the first initial features to obtain new first initial features and repeating this step until finally obtaining multimodal brain region activity signal features; inputting the first initial features and the second initial features into the function-to-structure conversion module, and performing weighted fusion on the features output by the function-to-structure conversion module and the second initial features to obtain new second initial features and repeating this step until finally obtaining multimodal structural features; inputting the multimodal brain region activity signal features into the orientation overall causal inference module to obtain multimodal effective connection matrices, and inputting the multimodal structural features into the structure decoding module to obtain reconstructed structural connection matrices.

[0008] As a further improvement of the present application, inputting the multi-modal effective connectivity matrix into the disease feature regression module for prediction, inputting the reconstructed structural connectivity matrix into the topological structure discriminator for prediction, and inputting the multi-modal brain region activity signal features into the time-space joint discriminator for prediction, including: inputting the multi-modal effective connectivity matrix into the disease feature regression module for prediction to obtain the disease state prediction probability; inputting the reconstructed structural connectivity matrix and the empirical structural connectivity matrix output by the preprocessing software template into the topological structure discriminator for prediction to obtain the probability that the reconstructed structural connectivity matrix is output by the correlation-aware dual-channel generation module or by the preprocessing software template; inputting the multi-modal brain region activity signal features and the empirical blood oxygen signal output by the preprocessing software template into the time-space joint discriminator for prediction to obtain the probability that the multi-modal brain region activity signal features are output by the correlation-aware dual-channel generation module or by the preprocessing software template.

[0009] As a further improvement of the present application, the time-space joint discriminator includes a time difference discriminator module and a space phase discriminator module. The time difference discriminator module is used to constrain the correlation-aware dual-channel generation module from the time continuity features of the brain region activity time series signal, and the space phase discriminator module constrains the correlation-aware dual-channel generation module from the spatial field distribution of the brain region activity signal.

[0010] As a further improvement of the present application, the loss function includes a disease feature regression loss, a topological adversarial loss, a topological perception loss, a time-space joint adversarial loss, and an attribution metric constraint loss;

[0011] The disease feature regression loss is used to guide the parameter update of the disease feature regression module and the correlation-aware dual-channel generation module, and is expressed as:

[0012]

[0013] Among them, represents the disease feature regression loss, A represents the multi-modal effective connectivity matrix, y represents the disease state, and p c (y|A) represents the disease state prediction probability, represents the expectation of the disease state probability predicted by the model under the true label distribution, and is used as the loss function to guide the model learning;

[0014] The topological adversarial loss is used to guide the parameter update of the topological structure discriminator and the correlation-aware dual-channel generation module, and is expressed as:

[0015]

[0016]

[0017] Among them, Represents the loss function for guiding the learning of the topological structure discriminator. Represents the loss function for guiding the learning of the generator by the topological structure discriminator. S represents the reconstructed structure connection matrix, S' represents the empirical structure connection matrix output by the preprocessing software template, and D top Represents the topological structure discriminator;

[0018] The topology-aware loss is used to guide the parameter update of the correlation-aware dual-channel generation module, and it is expressed as:

[0019]

[0020] Among them, Represents the topology-aware loss, ||·|| 2 Represents the Frobenius norm of the matrix, and λ represents a preset hyperparameter;

[0021] The time-space joint adversarial loss is used to guide the parameter update of the time difference discriminator module, the space phase discriminator module, and the correlation-aware dual-channel generation module, and it is expressed as:

[0022]

[0023]

[0024] Among them, Represents the loss function for guiding the learning of the time-space joint discriminator, Represents the loss function for guiding the learning of the generator by the time-space joint discriminator, and D tmp Represents the time difference discriminator module, and D spa Represents the space phase discriminator module;

[0025] The attribution metric constraint loss is used to guide the parameter update of the correlation-aware dual-channel generation module, and it is expressed as:

[0026]

[0027] Among them, Represents the attribution metric constraint loss, and B represents the multi-modal brain region activity signal feature.

[0028] As a further improvement of this application, the topological structure discriminator includes a multi-layer non-linear topology-aware network and a fully connected layer. The update formula of the multi-layer non-linear topology-aware network is expressed as:

[0029]

[0030] Among them, S represents the reconstructed structure connection matrix, D represents the weighted dispersion matrix corresponding to the reconstructed structure connection matrix, and F ( l )Represents the topological features of the l-th layer, F ( l +1) Represents the topological features of the (l + 1)-th layer, W ( l ) Is the learnable weight matrix in the l-th layer, b ( l ) Is the learnable non-linear bias in the l-th layer, and σ represents the sigmoid activation function.

[0031] To solve the above technical problems, another technical solution adopted by this application is: to provide a multi-modal brain network computing device with structural-functional association, which includes: an acquisition module for acquiring brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data; a fusion module for inputting the brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data into the interactive association perception fusion of the association perception dual-channel generation module of the brain disease prediction model to obtain multi-modal brain region activity signal features, multi-modal effective connection matrices, and reconstructed structural connection matrices; a prediction module for inputting the multi-modal effective connection matrices into the disease feature regression module of the brain disease prediction model for prediction, inputting the reconstructed structural connection matrices into the topological structure discriminator of the brain disease prediction model for prediction, and inputting the multi-modal brain region activity signal features into the time-space joint discriminator of the brain disease prediction model for prediction; an update module for reversely updating the association perception dual-channel generation module, the disease feature regression module, the topological structure discriminator, and the time-space joint discriminator according to the prediction results and a pre-constructed loss function.

[0032] To solve the above technical problems, another technical solution adopted by this application is: to provide a computer device, the computer device includes a processor and a memory coupled to the processor, and program instructions are stored in the memory. When the program instructions are executed by the processor, the processor executes the steps of the multi-modal brain network computing method with structural-functional association as described in any one of the above.

[0033] To solve the above technical problems, another technical solution adopted by this application is: to provide a storage medium storing program instructions capable of implementing the multi-modal brain network computing method with structural-functional association as described in any one of the above.

[0034] The beneficial effects of the present application are as follows: The multimodal brain network calculation method related to the structure and function of the present application cross-fuses functional magnetic resonance data and diffusion tensor imaging data of the brain by using the association perception dual-channel generation module of the brain disease prediction model to obtain multimodal brain region activity signal features, multimodal effective connection matrices, and reconstructed structural connection matrices. It realizes the non-linear multi-level fusion of multimodal heterogeneous and heterogeneous data, and then uses the multimodal brain region activity signal features to perform adversarial learning on the disease feature regression module, uses the multimodal effective connection matrix and the topological structure discriminator to perform adversarial learning, and uses the reconstructed structural connection matrix and the time-space joint discriminator to perform adversarial learning, constructing a multi-element collaborative generative adversarial strategy, comprehensively guiding the learning of the model from three aspects: the time continuity, spatial field distribution, and topological structure of the brain region activity time series signal, realizing the bidirectional constraint on the functional state and intrinsic structure of the multimodal effective connection, and greatly improving the accuracy, robustness, and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a schematic structural diagram of the brain disease prediction model according to an embodiment of the present invention;

[0036] Figure 2 is a schematic flowchart of the multimodal brain network calculation method related to the structure and function according to an embodiment of the present invention;

[0037] Figure 3 is a schematic structural diagram of the association perception dual-channel generation module according to an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of the functional modules of the multimodal brain network calculation device related to the structure and function according to an embodiment of the present invention;

[0039] Figure 5 is a schematic structural diagram of the computer device according to an embodiment of the present invention;

[0040] Figure 6 is a schematic structural diagram of the storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0042] The terms "first", "second", and "third" in this application are only used for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first", "second", and "third" may explicitly or implicitly include at least one of such features. In the description of this application, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined. All directional indications (such as up, down, left, right, front, back...) in the embodiments of this application are only used to explain the relative positional relationship, movement conditions, etc. between components in a specific posture (as shown in the drawings). If the specific posture changes, the directional indications will also change accordingly. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0043] Reference to "embodiment" in this context means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0044] Figure 1 is a schematic structural diagram of the brain disease prediction model according to an embodiment of the present invention. As Figure 1 shown, the brain disease prediction model includes an association-aware dual-channel generation module, a disease feature regression module, a topological structure discriminator, and a spatio-temporal joint discriminator. Among them, the association-aware dual-channel generation module is used to perform multi-level non-linear mutual conversion on the brain region image features extracted from brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data, and fuse the brain region function information and physical neuron connection information of the internal tissues of the brain. The disease feature regression module is used to predict the probability that a patient has the tested brain disease according to the multi-modal effective connection matrix output by the association-aware dual-channel generation module. The topological structure discriminator and the spatio-temporal joint discriminator are used to comprehensively guide the learning of the model from three aspects: the temporal continuity, spatial field distribution, and topological structure of the brain region activity time series signal, so as to realize the bidirectional constraint on the functional state and intrinsic structure of the multi-modal effective connection.

[0045] Figure 2 is a schematic flowchart of a multi-modal brain network calculation method for structural and functional association according to an embodiment of the present invention. It should be noted that if there are substantially the same results, the method of the present invention is not limited byFigure 2 subject to the process sequence shown. As Figure 2 shown, the multimodal brain network calculation method associated with this structure and function includes the steps:

[0046] Step S101: Obtain brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data.

[0047] Specifically, in this embodiment, pre-acquired brain functional magnetic resonance data (functional Magnetic Resonance Imaging, fMRI) and magnetic resonance diffusion tensor imaging data (Diffusion Tensor Imaging, DTI) are used as sample data to train a brain disease prediction model.

[0048] Step S102: Input the brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data into the correlation-aware dual-channel generation module for interactive correlation-aware fusion to obtain multimodal brain region activity signal features, multimodal effective connection matrices, and reconstructed structural connection matrices.

[0049] Specifically, please refer to Figure 3 , the correlation-aware dual-channel generation module includes a brain region feature extraction module, a structure-to-function conversion module, a function-to-structure conversion module, an orientation global causal inference module, and a structure decoding module. This correlation-aware dual-channel generation module adopts an alternating multi-layer interspersed structure that can extract complementary information at different scales and levels in heterogeneous data, and performs deep complementary information fusion in a non-linear iterative interaction manner to achieve the effect of efficiently fusing heterogeneous features

[0050] Further, step S102 specifically includes:

[0051] 1. Use the brain region feature extraction module to extract the first initial feature and the second initial feature from the brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data respectively.

[0052] Specifically, use the brain region feature extraction module to extract from the brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data respectively, and denote the first initial feature extracted from the brain functional magnetic resonance data as Denote the second initial feature extracted from the magnetic resonance diffusion tensor imaging data as where n represents the number of brain regions, and p and q respectively represent the dimension of the first initial feature vector and the dimension of the second initial feature vector for each brain region.

[0053] 2. Input the first initial feature and the second initial feature into the structure-to-function conversion module, and perform weighted fusion on the feature output by the structure-to-function conversion module and the first initial feature to obtain a new first initial feature and repeat this step until finally obtaining multimodal brain region activity signal features.

[0054] 3. Input the first initial feature and the second initial feature into the function-to-structure conversion module, and perform weighted fusion on the feature output by the function-to-structure conversion module and the second initial feature to obtain a new second initial feature, and repeat this step until the multi-modal structure feature is finally obtained.

[0055] In this embodiment, the first initial feature and the second initial feature are mutually transformed by the function-to-structure conversion module and the structure-to-function conversion module, and the above two conversion modules are implemented based on the correlation-aware Transformer. Among them, the output formula of the function-to-structure conversion module is:

[0056] F2S(X,Y) = Attention(q(X), k(Y), v(X||η·Y));

[0057] where η represents a hyperparameter with a default value of 0.1, the symbol || represents feature correlation aggregation, and q(·), k(·), v(·) are transformation functions composed of neural networks, which map X to the feature space with dimension respectively, map Y to the feature space with dimension respectively, and map X||η·Y to the feature space with dimension respectively.

[0058] Similarly, the output formula of the structure-to-function conversion module is:

[0059] S2F(X,Y) = Attention(q(Y), k(X), v(Y||η·X)).

[0060] where it maps X to the feature space with dimension respectively, maps Y to the feature space with dimension respectively, and maps Y||η·X to the feature space with dimension respectively.

[0061] In this embodiment, the calculation formula of the attention mechanism Attention is as follows:

[0062]

[0063] where

[0064] Specifically, when the first initial feature and the second initial feature are input into the structure-to-function conversion module, and the feature output by the structure-to-function conversion module is weighted and fused with the first initial feature to obtain a new first initial feature, with its weighting coefficient set to 0.1. Similarly, a new second initial feature is obtained by the function-to-structure conversion module. Thus, the first interactive correlation perception fusion is completed. Then, the above process is performed again on the new first initial feature and the new second initial feature. After several correlation perception fusions, finally, the multi-modal brain region activity signal feature T = (t 1 , t 2 , …, t n ) and the multi-modal structure feature D = (d 1 , d 2 , …, d n ) are obtained.

[0065] 4. Input the multi-modal brain region activity signal feature into the directional global causal inference module to obtain a multi-modal effective connectivity matrix, and input the multi-modal structure feature into the structure decoding module to obtain a reconstructed structural connectivity matrix.

[0066] The correlation perception dual-channel generation module of this embodiment uses an alternating multi-layer interspersed structure to achieve the overall non-linear fusion of features at different levels. Compared with traditional collaborative fusion or tower fusion, the alternating multi-layer interspersed structure proposed in this embodiment can extract complementary information at different scales and levels in heterogeneous data, and perform in-depth complementary information fusion in a non-linear and iterative interaction manner, achieving the effect of efficiently fusing heterogeneous features.

[0067] Step S103: Input the multi-modal effective connectivity matrix into the disease feature regression module for prediction, input the reconstructed structural connectivity matrix into the topological structure discriminator for prediction, and input the multi-modal brain region activity signal feature into the time-space joint discriminator for prediction.

[0068] Specifically, after obtaining the multi-modal effective connectivity matrix, the reconstructed structural connectivity matrix, and the multi-modal brain region activity signal feature, they are respectively input into the disease feature regression module, the topological structure discriminator, and the time-space joint discriminator for adversarial learning. Among them, the disease feature regression module is used to predict the prediction probability of suffering from the tested brain disease, and the topological structure discriminator and the time-space joint discriminator are used to comprehensively guide the learning of the model from three aspects: the time continuity, spatial field distribution, and topological structure of the brain region activity time series signal.

[0069] Furthermore, step S103 specifically includes:

[0070] 1. Input the multi-modal effective connectivity matrix into the disease feature regression module for prediction to obtain the disease state prediction probability.

[0071] Specifically, the disease feature regression module takes the multimodal effective connectivity matrix as input and outputs the predicted probability of the disease state to be tested. The disease feature regression module consists of a feature sensor, an information aggregation layer, a high-order feature extraction layer, an overall feature analysis layer, and a state probability prediction network, and finally obtains the probability of the disease state to be tested. By comparing with the known true disease state labels of the subjects, it guides the learning of the correlation-aware dual-channel generation module.

[0072] 2. Input the reconstructed structural connectivity matrix and the empirical structural connectivity matrix output by the preprocessing software template into the topological structure discriminator for prediction, and obtain the probability that the reconstructed structural connectivity matrix is output by the correlation-aware dual-channel generation module or by the preprocessing software template.

[0073] Specifically, the topological structure discriminator takes the reconstructed structural connectivity matrix and the empirical structural connectivity matrix output by the preprocessing software template as input, and outputs the probability that the reconstructed structural connectivity matrix is output by the correlation-aware dual-channel generation module or by the preprocessing software template. Among them, the topological structure discriminator includes a multi-layer non-linear topology perception network and a fully connected layer, and the update formula of the multi-layer non-linear topology perception network is expressed as:

[0074]

[0075] Among them, S represents the reconstructed structural connectivity matrix, D represents the weighted dispersion matrix corresponding to the reconstructed structural connectivity matrix, F ( l ) represents the topological feature of the l-th layer, F ( l +1) represents the topological feature of the (l + 1)-th layer, W ( l ) is the learnable weight matrix in the l-th layer, b ( l ) is the learnable non-linear bias in the l-th layer, σ represents the sigmoid activation function, and sigmoid is a library function in the deep learning framework pytorch.

[0076] Specifically, the multi-layer non-linear topology perception network uses the graph topology iteration technology to perceive the homology relationship in the structural network, and directly quantitatively calculates the topological features of each order from the structural connectivity matrix. Compared with the traditional method of using a multi-layer perceptron for feature extraction, the multi-layer non-linear topology perception network proposed in this embodiment focuses on the learning of topological features, excludes the interference of other irrelevant features, and can more comprehensively, systematically, and integrally depict the structural connection from the topological features.

[0077] 3. Input the multi-modal brain region activity signal features and the empirical blood oxygen signal output by the preprocessing software template into the time-space joint discriminator for prediction, and obtain the probability that the multi-modal brain region activity signal features are output by the associated perception dual-channel generation module or the preprocessing software template.

[0078] Among them, the time-space joint discriminator includes a time difference discriminator module and a space phase discriminator module. The time difference discriminator module consists of a temporal second-order difference layer, an oscillation fitting layer, a non-linear fusion layer, and a continuity analysis network; the space discriminator consists of a phase perception layer, a field strength detection layer, a field action path calculation layer, a non-linear fusion layer, and a field distribution prediction layer. The time difference discriminator module is used to constrain the associated perception dual-channel generation module from the time continuity characteristics of the brain region activity time series signal, and the space phase discriminator module constrains the associated perception dual-channel generation module from the space field distribution of the brain region activity signal, realizing the bidirectional constraint on the functional state and intrinsic structure of the multi-modal effective connection matrix.

[0079] Specifically, the time difference discriminator module and the space phase discriminator module take the multi-modal brain region activity signal features and the empirical blood oxygen signal output by the preprocessing software template as inputs, and output the probability that the multi-modal brain region activity signal features are output by the associated perception dual-channel generation module or the preprocessing software template.

[0080] Step S104: Update the associated perception dual-channel generation module, the disease feature regression module, the topological structure discriminator, and the time-space joint discriminator in reverse according to the predicted results and the pre-constructed loss function.

[0081] Among them, the loss function includes a disease feature regression loss, a topological adversarial loss, a topological perception loss, a time-space joint adversarial loss, and an attribution metric constraint loss.

[0082] The disease feature regression loss is constructed based on the Kullback-Leibler divergence and is used to guide the parameter update of the disease feature regression module and the associated perception dual-channel generation module. It is expressed as:

[0083]

[0084] Among them, represents the disease feature regression loss, A represents the multi-modal effective connection matrix, y represents the disease state. For example, taking Alzheimer's disease as an example, the disease state includes a healthy control group, mild cognitive impairment, late cognitive impairment, and Alzheimer's disease, and p c (y|A) represents the disease state prediction probability, represents the expectation of the disease state probability predicted by the model under the true label distribution, and is used as the loss function to guide the model learning;

[0085] The topological adversarial loss is used to guide the parameter update of the topological structure discriminator and the correlation-aware dual-channel generation module, and is expressed as:

[0086]

[0087]

[0088] Among them, represents the loss function that guides the learning of the topological structure discriminator, represents the loss function that guides the learning of the generator through the topological structure discriminator. The two together constitute the topological adversarial loss, aiming to learn the distribution of structural connections. S represents the reconstructed structural connection matrix, S′ represents the empirical structural connection matrix output by the preprocessing software template, and D top represents the topological structure discriminator.

[0089] In order to better capture the high-order topological structure differences between the reconstructed structural connections and the empirical structural connections, a topological awareness loss is designed in this embodiment to guide the parameter update of the correlation-aware dual-channel generation module, and is expressed as:

[0090]

[0091] Among them, represents the topological awareness loss, ||·|| 2 represents the Frobenius norm of the matrix, and λ represents a preset hyperparameter. This embodiment proposes a structure decoding module and a topological awareness loss. Compared with traditional adversarial learning methods that directly compare brain region features, the topological awareness loss depicts the low-order to high-order comprehensive topological differences between the reconstructed structural brain network and the empirical structural brain network obtained from the preprocessing software template. After the model training is completed, the model can accurately learn the topological structure features in the multimodal data without the annotation of professional physicians, thereby improving the accuracy of multimodal effective connectivity calculation.

[0092] The time-space joint adversarial loss is used to guide the parameter update of the time difference discriminator module, the space phase discriminator module, and the correlation-aware dual-channel generation module, and is expressed as:

[0093]

[0094]

[0095] Among them, represents the loss function that guides the learning of the time-space joint discriminator, It represents the loss function that guides the generator to learn through the time-space joint discriminator, aiming to enable the multi-modal brain region activity signals generated by the model to learn the time-frequency distribution of blood oxygenation level dependence extracted from fMRI data, D tmp It represents the time difference discrimination module, D spa It represents the spatial phase discrimination module. The time-space joint adversarial loss proposed in this embodiment integrates the time continuity feature and the spatial field distribution of the multi-modal brain region activity sequence into an organic whole. By introducing the second-order difference layer and the oscillation fitting layer to depict the continuity feature of the time series, and at the same time using the phase perception layer, the field strength detection layer, and the field action path calculation layer to depict the spatial field distribution of the brain region activity, the two jointly constrain the adversarial learning, solving the problem of low accuracy in the generation and extraction of time series in the traditional generative adversarial strategy.

[0096] It should be understood that the effective connectivity depicts the causal relationship between the activity signals of each brain region and satisfies the directional global constraint of the structural equation here ∈ i It represents noise. Based on this structural equation, this embodiment designs an attribution metric constraint loss to constrain the multi-modal effective connectivity matrix and the multi-modal brain region activity signal features learned by the model, and is used to guide the parameter update of the association perception dual-channel generation module, which is expressed as:

[0097]

[0098] Among them, It represents the attribution metric constraint loss, and B represents the multi-modal brain region activity signal features. In the association perception dual-channel generation module of this embodiment, based on the brain region activity signal structure equation theory in the effective connectivity, a directional global causal inference module and an attribution metric constraint loss are designed. Compared with the traditional effective connectivity calculation model that directly calculates the effective connectivity, the association perception dual-channel generation module proposed in this embodiment outputs both the multi-modal effective connectivity matrix and the multi-modal brain region activity signal features, and at the same time uses the attribution metric constraint loss to constrain the internal relationship between the two, making the output accuracy of the brain disease prediction model trained in this embodiment higher than that of the traditional prediction model and having stronger interpretability.

[0099] The multi-modal brain network calculation method related to structural functions in this embodiment cross-fuses functional magnetic resonance imaging (fMRI) data and diffusion tensor imaging (DTI) data of the brain by using the association-aware dual-channel generation module of the brain disease prediction model to obtain multi-modal brain region activity signal features, multi-modal effective connectivity matrices, and reconstructed structural connectivity matrices. It realizes the non-linear multi-level fusion of multi-modal heterogeneous and heterogeneous data, and then uses the multi-modal brain region activity signal features to perform adversarial learning on the disease feature regression module, uses the multi-modal effective connectivity matrix and the topological structure discriminator to perform adversarial learning, and uses the reconstructed structural connectivity matrix and the time-space joint discriminator to perform adversarial learning, constructing a multi-element collaborative generative adversarial strategy to comprehensively guide the learning of the model from three aspects: the time continuity, spatial field distribution, and topological structure of the brain region activity time series signal, realizing the bidirectional constraint on the functional state and intrinsic structure of multi-modal effective connectivity, and greatly improving the accuracy, robustness, and generalization ability of the model.

[0100] Further, after the multi-modal brain network related to structural functions is calculated, the brain disease prediction can be performed using this brain disease prediction. The method for predicting brain diseases using this brain disease prediction model includes:

[0101] 1. Obtain the functional magnetic resonance imaging (fMRI) data and diffusion tensor imaging (DTI) data of the patient's brain.

[0102] 2. Input the functional magnetic resonance imaging (fMRI) data and the diffusion tensor imaging (DTI) data into the association-aware dual-channel generation module of the brain disease prediction model for interactive association-aware fusion to obtain a multi-modal effective connectivity matrix.

[0103] 3. Input the multi-modal effective connectivity matrix into the disease feature regression module for prediction to obtain the predicted probability that the patient has a brain disease.

[0104] Figure 4 It is a schematic diagram of the functional modules of the multi-modal brain network calculation device related to structural functions in the embodiment of the present invention. As Figure 4 shown, the multi-modal brain network calculation device 20 related to structural functions includes an acquisition module 21, a fusion module 22, a prediction module 23, and an update module 24.

[0105] The acquisition module 21 is used to acquire the functional magnetic resonance imaging (fMRI) data and diffusion tensor imaging (DTI) data of the brain;

[0106] The fusion module 22 is used to input the functional magnetic resonance imaging (fMRI) data and the diffusion tensor imaging (DTI) data into the association-aware dual-channel generation module of the brain disease prediction model for interactive association-aware fusion to obtain multi-modal brain region signal features, multi-modal effective connectivity matrices, and reconstructed structural connectivity matrices;

[0107] The prediction module 23 is configured to input the multimodal effective connectivity matrix into the disease feature regression module of the brain disease prediction model for prediction, input the reconstructed structural connectivity matrix into the topological structure discriminator of the brain disease prediction model for prediction, and input the multimodal brain region signal features into the spatio-temporal joint discriminator of the brain disease prediction model for prediction;

[0108] The update module 24 is configured to inversely update the association-aware dual-channel generation module, the disease feature regression module, the topological structure discriminator, and the spatio-temporal joint discriminator according to the prediction results and a pre-constructed loss function.

[0109] Optionally, the association-aware dual-channel generation module includes a brain region feature extraction module, a structure-to-function conversion module, a function-to-structure conversion module, an orientation-based global causal inference module, and a structure decoding module.

[0110] Optionally, the fusion module 22 performs the operation of inputting the brain functional magnetic resonance data and the magnetic resonance diffusion tensor imaging data into the association-aware dual-channel generation module for interactive association-aware fusion to obtain multimodal brain region signal features, multimodal effective connectivity matrices, and reconstructed structural connectivity matrices, specifically including: using the brain region feature extraction module to respectively extract first initial features and second initial features from the brain functional magnetic resonance data and the magnetic resonance diffusion tensor imaging data; inputting the first initial features and the second initial features into the structure-to-function conversion module, and performing weighted fusion of the features output by the structure-to-function conversion module and the first initial features to obtain new first initial features and repeating this step until finally obtaining multimodal brain region signal features; inputting the first initial features and the second initial features into the function-to-structure conversion module, and performing weighted fusion of the features output by the function-to-structure conversion module and the second initial features to obtain new second initial features and repeating this step until finally obtaining multimodal structural features; inputting the multimodal brain region signal features into the orientation-based global causal inference module to obtain multimodal effective connectivity matrices, and inputting the multimodal structural features into the structure decoding module to obtain reconstructed structural connectivity matrices.

[0111] Optionally, the prediction module 23 performs operations of inputting the multimodal effective connectivity matrix into the disease feature regression module for prediction, inputting the reconstructed structural connectivity matrix into the topological discriminator for prediction, and inputting the multimodal brain region signal features into the time-space joint discriminator for prediction. Specifically, it includes: inputting the multimodal effective connectivity matrix into the disease feature regression module for prediction to obtain the disease state prediction probability; inputting the reconstructed structural connectivity matrix and the empirical structural connectivity matrix output by the preprocessing software template into the topological discriminator for prediction to obtain the probability that the reconstructed structural connectivity matrix is output by the correlation-aware dual-channel generation module or the preprocessing software template; inputting the multimodal brain region signal features and the empirical blood oxygen signal output by the preprocessing software template into the time-space joint discriminator for prediction to obtain the probability that the multimodal brain region signal features are output by the correlation-aware dual-channel generation module or the preprocessing software template.

[0112] Optionally, the time-space joint discriminator includes a time difference discriminator module and a space phase discriminator module. The time difference discriminator module is used to constrain the correlation-aware dual-channel generation module from the time continuity characteristics of the brain region activity time series signal, and the space phase discriminator module constrains the correlation-aware dual-channel generation module from the spatial field distribution of the brain region activity signal.

[0113] Optionally, the loss function includes a disease feature regression loss, a topological adversarial loss, a topological perception loss, a time-space joint adversarial loss, and an attribution metric constraint loss;

[0114] The disease feature regression loss is used to guide the parameter update of the disease feature regression module and the correlation-aware dual-channel generation module, and is expressed as:

[0115]

[0116] where represents the disease feature regression loss, A represents the multimodal effective connectivity matrix, y represents the disease state. For example, taking Alzheimer's disease as an example, the disease state includes healthy control group, mild cognitive impairment, late cognitive impairment, and Alzheimer's disease, p c (|) represents the disease state prediction probability, represents the expectation of the disease state probability predicted by the model under the true label distribution, and is used as the loss function to guide the model learning;

[0117] The topological adversarial loss is used to guide the parameter update of the topological discriminator and the correlation-aware dual-channel generation module, and is expressed as:

[0118]

[0119]

[0120] Among them, represents the loss function for guiding the learning of the topological structure discriminator, represents the loss function for guiding the learning of the generator by the topological structure discriminator. The two together constitute the topological adversarial loss, aiming to learn the distribution of structural connections. S represents the reconstructed structural connection matrix, S′ represents the empirical structural connection matrix output by the preprocessing software template, and D top represents the topological structure discriminator;

[0121] The topology-aware loss is used to guide the parameter update of the correlation-aware dual-channel generation module, and it is expressed as:

[0122]

[0123] Among them, represents the topology-aware loss, and ||·|| 2 represents the Frobenius norm of the matrix, and λ represents a preset hyperparameter;

[0124] The time-space joint adversarial loss is used to guide the parameter update of the time difference discriminator module, the space phase discriminator module, and the correlation-aware dual-channel generation module, and it is expressed as:

[0125]

[0126]

[0127] Among them, represents the loss function for guiding the learning of the time-space joint discriminator, represents the loss function for guiding the learning of the generator by the time-space joint discriminator, aiming to make the multi-modal brain region activity signals generated by the model learn the time-frequency distribution of blood oxygenation level dependence extracted from fMRI data, and D tmp represents the time difference discriminator module, and D spa represents the space phase discriminator module;

[0128] The attribution metric constraint loss is used to guide the parameter update of the correlation-aware dual-channel generation module, and it is expressed as:

[0129]

[0130] Among them, represents the attribution metric constraint loss, and B represents the multi-modal brain region signal features.

[0131] Optionally, the topological structure discriminator includes a multi-layer non-linear topology-aware network and a fully connected layer. The update formula of the multi-layer non-linear topology-aware network is expressed as:

[0132]

[0133] Among them, S represents the reconstructed structure connection matrix, D represents the weighted dispersion matrix corresponding to the reconstructed structure connection matrix, and F () represents the topological feature of the l-th layer, and F (+1) represents the topological feature of the (l + 1)-th layer, and W () is the learnable weight matrix in the l-th layer, and b () is the learnable non-linear bias in the l-th layer. σ represents the sigmoid activation function, and sigmoid is a library function in the deep learning framework pytorch.

[0134] For other details of the implementation technical solutions of each module in the above multi-modal brain network computing device, reference can be made to the description in the multi-modal brain network computing method in the above embodiments, which will not be elaborated here.

[0135] It should be noted that each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments.

[0136] Please refer to Figure 5 , Figure 5 which is the structural schematic diagram of the computer device according to the embodiment of the present invention. As Figure 5 shown, the computer device 30 includes a processor 31 and a memory 32 coupled to the processor 31. Program instructions are stored in the memory 32. When the program instructions are executed by the processor 31, the processor 31 executes the steps of the multi-modal brain network computing method described in any of the above embodiments.

[0137] Among them, the processor 31 can also be referred to as the CPU (Central Processing Unit, central processing unit). The processor 31 may be an integrated circuit chip with signal processing capabilities. The processor 31 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0138] Refer to Figure 6 , Figure 6Structural schematic diagram of the storage medium according to an embodiment of the present invention. The storage medium according to the embodiment of the present invention stores program instructions 41 capable of implementing the above multi-modal brain network calculation method. Among them, the program instructions 41 can be stored in the above storage medium in the form of a software product, including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, or computer devices such as computers, servers, mobile phones, and tablets.

[0139] In several embodiments provided by the present application, it should be understood that the disclosed computer devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0140] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units. The above is only the embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, is equally included in the patent protection scope of the present application.

Claims

1. A multimodal brain network calculation method for structural-functional association, characterized in that, it is applied to train a brain disease prediction model, and the brain disease prediction model includes an association-aware dual-channel generation module, a disease feature regression module, a topological structure discriminator, and a time-space joint discriminator; the method includes: obtaining brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data; inputting the brain functional magnetic resonance data and the magnetic resonance diffusion tensor imaging data into the association-aware dual-channel generation module for interactive association-aware fusion to obtain multimodal brain region activity signal features, a multimodal effective connection matrix, and a reconstructed structural connection matrix; inputting the multimodal effective connection matrix into the disease feature regression module for prediction, inputting the reconstructed structural connection matrix into the topological structure discriminator for prediction, and inputting the multimodal brain region activity signal features into the time-space joint discriminator for prediction; backward updating the association-aware dual-channel generation module, the disease feature regression module, the topological structure discriminator, and the time-space joint discriminator according to the prediction results and a pre-constructed loss function.

2. The multimodal brain network calculation method for structural-functional association according to claim 1, characterized in that, the association-aware dual-channel generation module includes a brain region feature extraction module, a structure-to-function conversion module, a function-to-structure conversion module, an orientation-based global causal inference module, and a structure decoding module.

3. The multimodal brain network calculation method for structural-functional association according to claim 2, characterized in that, the step of inputting the brain functional magnetic resonance data and the magnetic resonance diffusion tensor imaging data into the association-aware dual-channel generation module for interactive association-aware fusion to obtain multimodal brain region activity signal features, a multimodal effective connection matrix, and a reconstructed structural connection matrix includes: using the brain region feature extraction module to extract first initial features and second initial features from the brain functional magnetic resonance data and the magnetic resonance diffusion tensor imaging data respectively; inputting the first initial features and the second initial features into the structure-to-function conversion module, and performing weighted fusion on the features output by the structure-to-function conversion module and the first initial features to obtain new first initial features and repeating this step until finally obtaining the multimodal brain region activity signal features; inputting the first initial features and the second initial features into the function-to-structure conversion module, and performing weighted fusion on the features output by the function-to-structure conversion module and the second initial features to obtain new second initial features and repeating this step until finally obtaining multimodal structural features; inputting the multimodal brain region activity signal features into the orientation-based global causal inference module to obtain the multimodal effective connection matrix, and inputting the multimodal structural features into the structure decoding module to obtain the reconstructed structural connection matrix.

4. The multimodal brain network calculation method for structural-functional association according to claim 1, characterized in that, The steps of inputting the multi-modal effective connection matrix into the disease feature regression module for prediction, inputting the reconstructed structural connection matrix into the topological structure discriminator for prediction, and inputting the multi-modal brain region activity signal features into the time-space joint discriminator for prediction include: Inputting the multi-modal effective connection matrix into the disease feature regression module for prediction to obtain the disease state prediction probability; Inputting the reconstructed structural connection matrix and the empirical structural connection matrix output by the preprocessing software template into the topological structure discriminator for prediction to obtain the probability that the reconstructed structural connection matrix is output by the correlation-aware dual-channel generation module or the preprocessing software template; Inputting the multi-modal brain region activity signal features and the empirical blood oxygen signal output by the preprocessing software template into the time-space joint discriminator for prediction to obtain the probability that the multi-modal brain region activity signal features are output by the correlation-aware dual-channel generation module or the preprocessing software template.

5. The multi-modal brain network calculation method for structural and functional association according to claim 4, wherein, the time-space joint discriminator includes a time difference discriminator module and a space phase discriminator module. The time difference discriminator module is used to constrain the correlation-aware dual-channel generation module from the time continuity characteristics of the brain region activity time series signal, and the space phase discriminator module constrains the correlation-aware dual-channel generation module from the spatial field distribution of the brain region activity signal.

6. The multi-modal brain network calculation method for structural and functional association according to claim 5, wherein, the loss function includes a disease feature regression loss, a topological adversarial loss, a topological perception loss, a time-space joint adversarial loss, and an attribution metric constraint loss; the disease feature regression loss is used to guide the parameter update of the disease feature regression module and the correlation-aware dual-channel generation module, and is expressed as: Among them, represents the disease feature regression loss, A represents the multimodal effective connectivity matrix, y represents the disease state, and p c (|) represents the disease state prediction probability, represents the expectation of the disease state probability predicted by the model under the true label distribution, which is used as a loss function to guide the model learning; the topological adversarial loss is used to guide the parameter update of the topological structure discriminator and the correlation-aware dual-channel generation module, and is expressed as: Among them, represents the loss function for guiding the learning of the topological structure discriminator, represents the loss function for guiding the learning of the generator by the topological structure discriminator. S represents the reconstructed structure connection matrix, S ′ represents the empirical structure connection matrix output by the preprocessing software template, D top represents the topological structure discriminator; the topological perception loss is used to guide the parameter update of the correlation-aware dual-channel generation module, and is expressed as: Among them, represents the topology-aware loss, and ||·|| 2 represents the Frobenius norm of the matrix, and λ represents a preset hyperparameter; the time-space joint adversarial loss is used to guide the parameter update of the time difference discriminator module, the space phase discriminator module, and the correlation-aware dual-channel generation module, and is expressed as: Among them, represents the loss function for guiding the learning of the spatio-temporal joint discriminator, represents the loss function for guiding the learning of the generator by the spatio-temporal joint discriminator, D tmp represents the time difference discriminator module, D spa represents the spatial phase discriminator module; the attribution metric constraint loss is used to guide the parameter update of the correlation-aware dual-channel generation module, and is expressed as: Among them, represents the attribution metric constraint loss, and B represents the multi-modal brain region activity signal feature.

7. The multi-modal brain network calculation method for structural and functional association according to claim 1, wherein, the topological structure discriminator includes a multi-layer non-linear topological perception network and a fully connected layer, and the update formula of the multi-layer non-linear topological perception network is expressed as: Among them, S represents the reconstructed structure connection matrix, D represents the weighted dispersion matrix corresponding to the reconstructed structure connection matrix, and F () represents the topological feature of the l-th layer, and F (+1) represents the topological feature of the (l + 1)-th layer. W () is the learnable weight matrix in the l-th layer, and b () is the learnable non-linear bias in the l-th layer. σ represents the sigmoid activation function.

8. A multi-modal brain network calculation device for structural and functional association, wherein, it includes: an acquisition module for acquiring brain functional magnetic resonance data and magnetic resonance diffusion tensor imaging data; A fusion module, configured to input the brain functional magnetic resonance data and the magnetic resonance diffusion tensor imaging data into an interactive association perception fusion module of a brain disease prediction model, so as to obtain multi-modal brain region activity signal features, a multi-modal effective connection matrix, and a reconstructed structural connection matrix; A prediction module, configured to input the multi-modal effective connection matrix into a disease feature regression module of the brain disease prediction model for prediction, input the reconstructed structural connection matrix into a topological structure discriminator of the brain disease prediction model for prediction, and input the multi-modal brain region activity signal features into a time-space joint discriminator of the brain disease prediction model for prediction; An update module, configured to inversely update the association perception dual-channel generation module, the disease feature regression module, the topological structure discriminator, and the time-space joint discriminator according to the prediction result and a pre-constructed loss function.

9. A computer device, characterized in that, the computer device includes a processor and a memory coupled to the processor, and program instructions are stored in the memory. When the program instructions are executed by the processor, the processor is caused to execute the steps of the multi-modal brain network calculation method related to structural function association according to any one of claims 1-7.

10. A storage medium, characterized in that, it stores program instructions capable of implementing the multi-modal brain network calculation method related to structural function association according to any one of claims 1-7.

Images