Disease classification method and system fusing multi-mode brain function connection space-time and causal features

Abstract

The invention discloses a disease classification method fusing multi-modal brain function connection space-time and causal features, which comprises the following steps: acquiring brain image data of a subject to construct a data set for training; constructing a deep learning network model which comprises a time sequence coding module, a cross attention fusion module, a modal fusion module and a graph convolution prediction module; and training the deep learning network model by using the data set to obtain a prediction model for disease classification. The invention also provides a disease classification system. According to the method provided by the invention, the time sequence features, the relevance features and the causality features are efficiently fused, and finally accurate disease classification is realized.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing, and in particular relates to a disease classification method and system that integrates spatiotemporal and causal features of multimodal brain functional connectivity. Background Technology

[0002] Functional magnetic resonance imaging (fMRI), as a mature non-invasive brain imaging technique, has been widely used in the study of brain functional networks and the exploration of the pathological mechanisms of neuropsychiatric diseases. By analyzing fMRI time-series data, various connectivity matrices describing brain region interaction patterns can be constructed, such as the functional connectivity matrix (FC), Pearson correlation matrix, Spearman correlation matrix, and other correlation measures, as well as causality measures such as Granger causality matrix and nonlinear Granger causality matrix (NGC).

[0003] However, existing technologies generally face the following technical bottlenecks when using these multimodal brain connectivity data for disease classification: 1. Defects in multimodal information fusion mechanisms: Traditional methods often use simple feature splicing or linear weighting to fuse different connectivity matrices. This approach cannot adaptively capture the complex nonlinear dependencies between modalities, resulting in the failure to fully utilize the complementary advantages of multimodal information. 2. Insufficient spatiotemporal feature co-modeling ability: Most models tend to process fMRI time series (temporal information) or static connectivity matrices (spatial topological information) in isolation, lacking an effective mechanism to deeply and co-model dynamic temporal dependencies with static spatial connectivity patterns. 3. Limited model generalization and robustness: Due to significant differences in acquisition protocols and subject groups among different fMRI datasets, existing models typically have poor cross-dataset generalization ability, limiting their reliability in practical clinical applications.

[0004] Patent document CN119517371A discloses a deep joint learning diagnostic method for Alzheimer's disease based on multimodal feature fusion. It aims to construct an end-to-end deep joint learning diagnostic method based on residual neural networks and attention mechanisms, which automatically learns feature information and location information related to AD to obtain the classification basis for diagnosis. The experiment includes three levels of classification: disease diagnosis based on AD and HC, AD and MCI, and MCI and HC.

[0005] Patent document CN113616184A discloses a brain network modeling and individual prediction method based on multimodal magnetic resonance imaging (fMRI), including: constructing a T1 structural covariant brain network, a DTI white matter brain network, an fMRI functional brain network, and brain network analysis and calculation. Based on an established sample database of a dementia cohort in the Chinese population, it can simultaneously construct multimodal brain networks and calculate various related brain network indicators, unifying the output format of each modality to facilitate the application of multimodal brain networks. Information features are selected from the feature indicators of the multimodal brain networks to construct an automatic discriminant analysis model for mild cognitive impairment based on support vector machines and a brain age prediction model. Summary of the Invention

[0006] The purpose of this invention is to provide a disease classification method and system that integrates spatiotemporal and causal features of multimodal brain functional connectivity. This method achieves accurate disease classification by efficiently integrating time series features, correlation features, and causal features.

[0007] To achieve the first objective of this invention, the following technical solution is provided: a disease classification method that integrates spatiotemporal and causal features of multimodal brain functional connectivity, comprising the following steps: Brain imaging data of subjects were acquired, and data were extracted from each brain region in the brain imaging data in time series to construct brain functional connectivity data including functional connectivity matrix, multiple correlation connectivity matrix and multiple causal connectivity matrix. Brain functional connectivity data are labeled according to disease type, and brain imaging data, labels and brain functional connectivity data are combined into a dataset. Construct a deep learning network model, including a time series encoding module, a cross-attention fusion module, a modality fusion module, and a graph convolution prediction module; The time series encoding module is used to encode the input brain image data and extract the features of the encoding results in the time dimension to obtain the corresponding brain region embedding features. The cross-attention fusion module includes an association feature fusion unit and a causal feature fusion unit. The association feature fusion unit takes multiple association connection matrices as input to fuse and obtain corresponding fused association features. The causal feature fusion unit takes multiple causal connection matrices as input to fuse and obtain corresponding fused causal features. The modality fusion module performs feature fusion based on brain region embedding features, fusion association features, and fusion causal features to output comprehensive node features; The graph convolution prediction module uses the input comprehensive node features as nodes of the graph, constructs the adjacency matrix corresponding to the graph based on the connection matrix in the input brain functional connectivity data, and outputs the prediction result through multi-layer graph convolution and brain region pooling operations. A deep learning network model is trained using a dataset to obtain a predictive model for disease classification.

[0008] This invention designs a parallel, multi-branch modular architecture and introduces a cross-attention mechanism to deeply integrate the temporal, correlational, and causal information of fMRI data, thereby significantly improving the accuracy of neuropsychiatric disease classification and the generalization ability of the model.

[0009] Specifically, the various correlation matrices include the Pearson correlation matrix and the Spearman correlation matrix.

[0010] Specifically, the various causal connection matrices include Granger causal connection matrices and nonlinear Granger causal connection matrices.

[0011] Specifically, the process of constructing the brain functional connectivity data is as follows: Time-series data from brain imaging were collected and processed by mainstream fMRI pipeline preprocessing software to generate time-series data for 246 and 400 brain regions. Functional connectivity matrix, nonlinear Granger causal connectivity matrix, Granger causal connectivity matrix, Pearson correlation matrix, and Spearman correlation matrix were calculated based on the time series data. The data was in MAT file format and included subject ID information.

[0012] Specifically, the fMRI preprocessing pipeline includes the following sub-steps: (1) Head motion correction: Rigid body transformation algorithm is used to correct head translation (threshold 0.5mm) and rotation (threshold 0.5) during the scanning process to remove motion artifacts; (2) Time layer correction: Based on the slice acquisition order, the time points of different slices are aligned by linear interpolation to eliminate time delay differences; (3) Spatial normalization: fMRI images were registered to the MNI152 standard brain template (3mm×3mm×3mm), and nonlinear transformation was used to reduce individual brain structural heterogeneity; (4) Signal denoising: Physiological noise and low-frequency drift are removed by regressing cerebrospinal fluid, white matter and global signals, combined with 0.01-0.1Hz bandpass filtering; (5) Brain region division: Based on the BN_Atlas246 or Schaefer400 brain region template, the whole brain is divided into 246 or 400 regions of interest (ROIs), and the average time series of each ROI is extracted and used as the time series input to the time series encoding module.

[0013] Specifically, the data preprocessing includes extracting subject IDs from filenames using regular expressions, and performing modal alignment of fMRI time series, FC matrix, Granger, NGC, Pearson, and Spearman matrices by subject ID to ensure that subject IDs in the control group and patient group are unique and without duplication; the data standardization uses the Standard Scaler method based on the mean and standard deviation of the training set, as shown in the following expression: ; in, and These are the mean and standard deviation of the data, respectively.

[0014] Specifically, the time series encoding module is constructed using a 3-layer Transformer encoder, with each layer containing 2 attention heads and a multilayer perceptron with a hidden layer dimension of 128.

[0015] Specifically, the fusion process of the cross-attention fusion module is as follows: Flatten the output function connection matrix into the corresponding one-dimensional feature vector; For each feature fusion unit, construct the corresponding Q, K, and V, and perform attention calculations to obtain attention scores and weights; Based on the calculated attention score and weight, the one-dimensional feature vectors in the same feature fusion unit are weighted and fused to output the corresponding fused feature.

[0016] Specifically, the adjacency matrix is ​​based on at least one connection matrix in the input brain functional connectivity data. The cosine similarity between any two brain region embedding vectors in each connection matrix is ​​calculated to obtain the functional connectivity strength between brain regions and output as the adjacency matrix.

[0017] Specifically, one method for generating the adjacency matrix is ​​as follows: based on the brain region embedding features ([B, R, D]) output by the time series encoding module, an FCMatrix (shape [B, R, R]) is generated by calculating the cosine similarity between any two brain region embedding vectors. The expression is: in, Let i be the embedding vector of the i-th brain region. The functional connectivity strength between the i-th and j-th brain regions.

[0018] Specifically, the process of the multi-layer graph convolution and brain region pooling operations is as follows: The integrated node features and adjacency matrix are used as input to a two-layer graph convolutional network. The first layer of the graph convolutional network performs feature aggregation to output the corresponding predicted value and calculate the brain region importance score of the corresponding brain region. The second-layer graph convolutional network then performs Top-50% brain region pooling on the output predictions before inputting them into the 2-layer MLP to output the disease classification probability.

[0019] Specifically, the feature aggregation formula is as follows: ; in, This is the output feature matrix of the first layer of the graph convolutional network. It is an adjacency matrix. The input is the comprehensive node feature matrix. This is the learnable weight matrix of the first layer of the graph convolutional network. This represents the bias vector of the first layer of the graph convolutional network. For activation function, This indicates batch normalization operation.

[0020] Brain region importance scores are calculated using the first-layer weight matrix of the GCN, expressed as: in, Let be the GCN weight vector corresponding to the i-th brain region.

[0021] Specifically, during the training process, multiple loss functions are used to train the deep learning network model. These multiple loss functions include cross-entropy loss and sparsity loss, and their expressions are as follows: ; in, Represents cross-entropy loss, Indicates sparsity loss. These represent the weight parameters for various loss functions.

[0022] Specifically, the cross-entropy loss expression is as follows: in, For the sample size, For category weights, For real labels (unique hot encoding). To predict probabilities.

[0023] Specifically, the expression for the sparsity loss is as follows: The weight is 0.5 × 10. -4This is used to balance classification performance and feature sparsity. Represents the functional connection matrix. This represents a nonlinear Granger causal connection matrix.

[0024] To achieve the second objective of this invention, the following technical solution is provided: a disease classification system for performing the steps of the above-described disease classification method that integrates spatiotemporal and causal features of multimodal brain functional connectivity.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: By organically combining the Transformer encoder, which excels at capturing temporal dynamics, with the graph convolutional network, which is adept at learning spatial topology, a deep and collaborative modeling of both the spatiotemporal dimensions of fMRI data was achieved. Furthermore, a parallel cross-attention fusion mechanism was utilized to adaptively learn and deeply fuse complementary features of the two types of brain functional connections: correlation and causation. This significantly improved the model's generalization ability and robustness when facing multi-source heterogeneous data (such as datasets from different sites). As a result, not only was a high-precision classification of neuropsychiatric diseases achieved, but the contribution of each brain region to the disease classification could also be quantitatively analyzed, providing interpretable decision support for locating key biomarkers related to diseases. Attached Figure Description

[0026] Figure 1 This is a flowchart of the disease classification method that integrates spatiotemporal and causal features of multimodal brain functional connectivity provided in this embodiment; Figure 2 This is a schematic diagram of the cross-attention feature fusion module provided in this embodiment; Figure 3 This is a schematic diagram of the modal fusion module provided in this embodiment; Figure 4 This is a schematic diagram of the graph convolution prediction module provided in this embodiment. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0028] like Figure 1 As shown in this embodiment, a disease classification method integrating spatiotemporal and causal features of multimodal brain functional connectivity is provided. The specific steps are as follows: A dataset was constructed, comprising brain functional connectivity data based on functional magnetic resonance imaging (fMRI). The brain functional connectivity data included functional connectivity matrices (FC), nonlinear Granger causal connectivity matrices (NGC), Granger causal connectivity matrices (Granger), Pearson correlation matrices, and Spearman correlation matrices. The data was derived from time-series data of multiple brain regions and standardized connectivity matrices were generated through an fMRI preprocessing pipeline and standardization. Construct a deep learning network model, which includes a time series encoding module, a cross-attention fusion module, a modality fusion module, and a graph convolution prediction module.

[0029] The time series encoding module is used to encode the input fMRI time series data, capture the time dimension dependency through the Transformer encoder, and generate brain region embedding features. The cross-attention fusion module includes an association feature fusion unit and a causal feature fusion unit. The association feature fusion unit takes the Pearson correlation matrix and the Spearman correlation matrix as input, performs cross-attention fusion, and outputs fused association features. The causal feature fusion unit takes the Granger causal connection matrix and NGC as input, performs cross-attention fusion, and outputs fused causal features. The modality fusion module is used to fuse the fusion correlation features, fusion causal features, and brain region embedding features to generate comprehensive node features for subsequent prediction. The graph convolution prediction module uses the comprehensive node features as nodes of the graph and constructs an adjacency matrix of the graph based on at least one functional connectivity data (e.g., the original functional connectivity matrix FC). Through multi-layer graph convolution and brain region pooling operations, it outputs disease classification results and corresponding brain region importance scores. The deep learning network model was trained using a dataset with the Adam optimizer and a learning rate of 1.0 × 10⁻⁶. -4 The weight decay is set to 1.0 × 10. -4 The model was trained for 500 rounds; five-fold cross-validation was used, and the model was optimized through cross-entropy loss and sparsity loss to obtain a model for disease classification. The brain function connectivity data to be classified is input into the trained model, and the corresponding disease classification results are output.

[0030] like Figure 2As shown, this is the cross-attention fusion module mentioned in this embodiment. This module consists of three core components cascaded together: an association feature fusion unit, a causal feature fusion unit, and a cross-modal cross-attention unit. in Figure 2 In the diagram, (a) represents the correlation feature fusion processor, which flattens the input two-dimensional Pearson correlation matrix and Spearman correlation matrix (shape [B, R, R]) into one-dimensional feature vectors (shape [B, R]). 2 The flattened Pearson matrix vector is used as the query vector Q, and the flattened Spearman matrix vector is used as the key vector K and value vector V. The attention score and weights are calculated using the following formula: in, , where is the dimension of the key vector, used for scaling to avoid gradient vanishing.

[0031] The fused association features are obtained by weighted summation of attention weights and value vector V, and their expression is as follows: The final output is a fused correlation feature.

[0032] in Figure 2 (b) in the diagram represents the causal feature fusion unit, which employs the same architecture as the correlation feature fusion unit to integrate two types of causal connectivity information. Its inputs are the Granger causal connectivity matrix and the nonlinear Granger (NGC) causal connectivity matrix. After feature flattening, the Granger vector is used as the query vector Q, and the NGC vector is used as the key vector K and value vector V. The same attention calculation and feature fusion formulas are applied, ultimately outputting fused causal features.

[0033] in Figure 2 (c) in the diagram represents cross-modal attention. This module is responsible for performing a higher level of deep fusion of the initially fused association and causal modal information. Using the fused association features output from the first step as the new query vector Q, and the fused causal features output from the second step as the new key vector K and value vector V, the attention calculation and feature fusion formula described above are executed again to obtain the comprehensive fused features.

[0034] like Figure 3 The diagram shown is a schematic of the modal fusion module provided in this embodiment. Figure 3(a) represents the generation of the time-series encoding and functional connectivity matrix. The input to this module is the raw fMRI time series. The time-series encoder performs deep processing on the input fMRI time series to effectively capture the complex dependencies of brain region signals in the time dimension. This encoder internally generates feature representations such as query vector Q, key vector K, and value vector V, and finally outputs the embedding features of each brain region. Based on these brain region embedding features output by the Transformer encoder, the functional connectivity matrix is ​​dynamically generated by calculating the similarity between any two brain region embedding vectors. This FC matrix reflects the correlation strength between brain regions.

[0035] in Figure 3 (b) represents causal feature fusion. The input to this module is a linear Granger causal connection matrix and a nonlinear Granger causal connection matrix. Through a cross-attention mechanism, the linear Granger causal matrix is ​​mapped to a query vector Q, and the nonlinear Granger causal matrix is ​​mapped to a key vector K and a value vector V. First, the attention weights are obtained by the dot product of Q and K. Then, these weights are applied to V for weighted summation, and finally, the fused causal matrix is ​​output, completing the extraction and integration of causal dimension features.

[0036] in Figure 3 (c) represents the cross-attention fusion of the functional connectivity matrix and the causal matrix. This module aims to deeply interact and fuse the spatial functional connectivity information generated by time-series encoding with the fused causal dependency information to generate a more comprehensive feature representation. The functional connectivity matrix generated by the time-series encoding module is used as the query vector Q, and the fused causal matrix output by the causal feature fusion unit is used as the key vector K and the value vector V. Attention weights are calculated by the dot product of Q and K, and then V is weighted and aggregated to finally generate a comprehensive fused feature. This feature simultaneously covers the temporal-spatial association and causal dependency information of functional connectivity, providing comprehensive and discriminative feature support for subsequent disease classification tasks.

[0037] like Figure 4 As shown, this is the graph convolutional prediction module mentioned in this embodiment. This module employs a multi-layer graph convolutional network architecture, using comprehensive node features as the node attributes of the graph, and a functional connectivity matrix or a critical matrix dynamically generated based on brain region embedding features as the graph's topology. Through effective feature aggregation and high-level feature extraction via multiple GCN layers, combined with brain region pooling operations, it aims to identify key brain region features related to diseases from rich brain functional connectivity information. Finally, after processing by multi-layer perceptron layers, it provides interpretable support for clinical auxiliary diagnosis.

[0038] More specifically, before performing graph convolution operations, the graph convolution prediction module first constructs an adjacency matrix of the graph. One way to generate this adjacency matrix is ​​as follows: based on the brain region embedding features ([B, R, D]) output by the time series encoding module, an FC Matrix (shape [B, R, R]) is generated by calculating the cosine similarity between any two brain region embedding vectors. The expression is: in, Let i be the embedding vector of the i-th brain region. The functional connectivity strength between the i-th and j-th brain regions.

[0039] A two-layer graph convolutional network (GCN) is used, combining batch normalization (BatchNorm) and the LeakyReLU activation function (negative slope 0.2). The first layer of the GCN takes the integrated embedding features (F_combine, [B, R, D]) and the adjacency matrix (A, [B, R, R]) as input, and the feature aggregation formula is as follows: The second-layer GCN output features are pooled in the top-50% of brain regions (selecting important brain regions based on the L2 norm of feature weights) and then input into a 2-layer MLP (hidden layer dimension 64), outputting disease classification probabilities (shape [B, 2], corresponding to patients and controls); the brain region importance score is calculated using the weight matrix of the first layer of the GCN, expressed as: in, Let be the GCN weight vector corresponding to the i-th brain region.

[0040] During training, the deep learning network model was trained using the dataset, employing the Adam optimizer with a learning rate of 1.0 × 10⁻⁶. -4 The weight decay is set to 1.0 × 10. -4 The model was trained for 500 rounds; five-fold cross-validation was used, and the model was optimized through multiple loss functions to obtain a predictive model for disease classification.

[0041] These various loss functions include cross-entropy loss and sparsity loss, and the overall expression is as follows: ; The cross-entropy loss expression is as follows: ; in, For the sample size, For category weights, For real labels (unique hot encoding). To predict probabilities; The expression for the sparsity loss is as follows: The weight is 0.5 × 10. -4 This is used to balance classification performance with feature sparsity.

[0042] This embodiment also provides a disease classification system for performing the steps of the disease classification method that integrates spatiotemporal and causal features of multimodal brain functional connectivity provided in the above embodiments.

[0043] This invention utilizes multimodal fusion and graph convolutional networks to fully leverage the spatial and temporal information of brain functional connectivity data, significantly improving the accuracy and robustness of disease classification. It is applicable to the diagnosis of neuropsychiatric disorders, including but not limited to non-suicidal self-harm behaviors.

Claims

1. A disease classification method integrating spatiotemporal and causal features of multimodal brain functional connectivity, characterized in that, Includes the following steps: Brain imaging data of subjects were acquired, and data were extracted from each brain region in the brain imaging data in time series to construct brain functional connectivity data including functional connectivity matrix, multiple correlation connectivity matrix and multiple causal connectivity matrix. Brain functional connectivity data are labeled according to disease type, and brain imaging data, labels and brain functional connectivity data are combined into a dataset. Construct a deep learning network model, including a time series encoding module, a cross-attention fusion module, a modality fusion module, and a graph convolution prediction module; The time series encoding module is used to encode the input brain image data and extract the features of the encoding results in the time dimension to obtain the corresponding brain region embedding features. The cross-attention fusion module includes an association feature fusion unit and a causal feature fusion unit. The association feature fusion unit takes multiple association connection matrices as input to fuse and obtain corresponding fused association features. The causal feature fusion unit takes multiple causal connection matrices as input to fuse and obtain corresponding fused causal features. The modality fusion module performs feature fusion based on brain region embedding features, fusion association features, and fusion causal features to output comprehensive node features; The graph convolution prediction module uses the input comprehensive node features as nodes of the graph, constructs the adjacency matrix corresponding to the graph based on the connection matrix in the input brain functional connectivity data, and outputs the prediction result through multi-layer graph convolution and brain region pooling operations. A deep learning network model is trained using a dataset to obtain a predictive model for disease classification.

2. The disease classification method based on the fusion of spatiotemporal and causal features of multimodal brain functional connectivity according to claim 1, characterized in that, The various correlation matrices include the Pearson correlation matrix and the Spearman correlation matrix.

3. The disease classification method based on the fusion of spatiotemporal and causal features of multimodal brain functional connectivity according to claim 1, characterized in that, The various causal connection matrices include Granger causal connection matrices and nonlinear Granger causal connection matrices.

4. The disease classification method based on the fusion of spatiotemporal and causal features of multimodal brain functional connectivity according to claim 1, characterized in that, The time series encoding module is constructed using a 3-layer Transformer encoder, with each layer containing 2 attention heads and a multilayer perceptron with a hidden layer dimension of 128.

5. The disease classification method based on the fusion of spatiotemporal and causal features of multimodal brain functional connectivity according to claim 1, characterized in that, The fusion process of the cross-attention fusion module is as follows: Flatten the output function connection matrix into the corresponding one-dimensional feature vector; For each feature fusion unit, construct the corresponding Q, K, and V, and perform attention calculations to obtain attention scores and weights; Based on the calculated attention score and weight, the one-dimensional feature vectors in the same feature fusion unit are weighted and fused to output the corresponding fused feature.

6. The disease classification method based on the fusion of spatiotemporal and causal features of multimodal brain functional connectivity according to claim 1, characterized in that, The adjacency matrix is ​​based on at least one connection matrix in the input brain functional connectivity data. The cosine similarity between any two brain region embedding vectors in each connection matrix is ​​calculated to obtain the functional connectivity strength between brain regions and output as the adjacency matrix.

7. The disease classification method based on the fusion of spatiotemporal and causal features of multimodal brain functional connectivity according to claim 1, characterized in that, The process of multi-layer graph convolution and brain region pooling operations is as follows: The integrated node features and adjacency matrix are used as input to a two-layer graph convolutional network. The first layer of the graph convolutional network performs feature aggregation to output the corresponding predicted value and calculate the brain region importance score of the corresponding brain region. The second-layer graph convolutional network then performs Top-50% brain region pooling on the output predictions before inputting them into the 2-layer MLP to output the disease classification probability.

8. The disease classification method based on the fusion of spatiotemporal and causal features of multimodal brain functional connectivity according to claim 1, characterized in that, During training, multiple loss functions are used to train the deep learning network model, including cross-entropy loss and sparsity loss.

9. A disease classification system, characterized in that, Steps for performing the disease classification method that integrates spatiotemporal and causal features of multimodal brain functional connectivity as described in any one of claims 1 to 8.

Images