Depression image classification method based on multi-modal dynamic graph convolutional neural network

Abstract

The application discloses a depression image classification method based on a multi-modal dynamic graph convolutional neural network, belongs to the field of medical image processing and artificial intelligence, and utilizes a sliding time window method to divide a time sequence after functional magnetic resonance data preprocessing into multiple time windows, constructs a functional connection matrix of each window, utilizes multiple spatial attention contrast networks to capture time information across time windows, and obtains a high-order dynamic functional brain network; and utilizes a cross-modal graph neural network and cross-modal knowledge distillation to realize complementary information transmission and mode fusion between modes. The application can fully utilize dynamic high-order information of a multi-modal brain network to realize effective classification of depression.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of medical image processing and artificial intelligence, and particularly relates to a depression image classification method based on a multi-modal dynamic graph convolutional neural network. BACKGROUND

[0002] With the development of artificial intelligence and neural imaging, the analysis and processing of magnetic resonance imaging data by using neural imaging analysis technology and brain network modeling can realize early diagnosis and classification of depression, can assist doctors in qualitatively or even quantitatively analyzing depression-specific biomarkers, and thus improve the accuracy and reliability of depression diagnosis and classification.

[0003] Brain network analysis models the brain based on the idea of graph theory, reveals the complex relationship between brain regions, and has become an important tool for neural science research and neural system disease diagnosis. Previous studies have shown that changes in the connection mode of brain networks are associated with depression. Therefore, brain network analysis can provide an effective means for depression diagnosis and biomarker identification.

[0004] The current commonly used brain network analysis methods mainly include functional brain network analysis and structural brain network analysis. Most current depression classification based on functional brain networks only considers static functional connections, ignores the dynamic characteristics of connections fluctuating over time, that is, is limited to extracting isolated features in discrete time slices, and is difficult to fully capture high-order spatiotemporal topological features in continuous time intervals. On the other hand, most depression classification methods based on brain networks only consider single-modal brain networks, and do not consider the important influence of multi-modal complementary information in brain network analysis. SUMMARY

[0005] In order to solve the problems of insufficient classification accuracy and poor robustness caused by insufficient consideration of multi-modal complementary information and dynamic spatiotemporal characteristics in some existing depression classification methods based on brain networks, the application provides a depression image classification method based on a multi-modal dynamic graph convolutional neural network, which can fully utilize the dynamic high-order information of multi-modal brain networks to realize effective classification of depression.

[0006] The technical scheme adopted by the application to solve its technical problems is:

[0007] A depression image classification method based on a multi-modal dynamic graph convolutional neural network, comprising the following steps:

[0008] Step 1: The time series of functional magnetic resonance imaging (fMRI) data is divided into a plurality of non-overlapping time windows by using a sliding window method, and a functional connection matrix between brain regions is generated by calculating the correlation of the time series in each window, so as to quantify the functional connectivity between brain regions;

[0009] Step two, high-order spatio-temporal topology features are extracted by multi-channel spatial attention contrast network;

[0010] Step three, contrast learning constraints are introduced, and the fusion of spatio-temporal features is realized through self-attention mechanism and LSTM, and high-order dynamic functional network DBNs (Dynamic Brain Networks) are calculated through scaled dot-product attention;

[0011] Step four, the input DBNs and structural brain networks are independently encoded to generate modality-specific embedding features;

[0012] Step five, the DBNs and structural modality features are respectively aggregated on the dynamic graph structure through bilateral graph convolution;

[0013] Step six, auxiliary classifiers are used for DBNs modality, structural modality and multi-modal joint features respectively, and single modality prediction probability distribution is outputted, then the consistency of multi-modal prediction and single modality prediction is constrained through KL divergence; the model adopts gradual weight adjustment.

[0014] Further, in the step three, the intra-window contrast loss is maximized by maximizing the similarity of features in the same time window in different enhanced views, while minimizing the similarity between different brain regions; these contrast constraints enable the model to learn spatio-temporal features with discriminative significance; the embedding features of each window are mapped to Query, Key and Value matrices.

[0015] Still further, in the step four, for the DBNs corresponding features and the structural network corresponding features , the cross-modal mapping matrix of the model is calculated; then is normalized to become a double random matrix, and finally the cross-modal correlation weight matrix is obtained, the normalized matrix is used for cross-modal feature mapping, which reflects the weight distribution when cross-modal fusion.

[0016] Still further, in the step six, the auxiliary classifier of DBNs modality receives the graph-convoluted features , the auxiliary classifier of structural modality receives the graph-convoluted features , and the auxiliary classifier of multi-modal joint features receives the graph-convoluted features , and the three modalities output single modality prediction probability distribution , and The auxiliary classifier is composed of a full connection layer and a SoftMax function.

[0017] Preferably, in step six, the model adopts a gradual weight adjustment process: the weight of distillation loss increases linearly with the training round , , approaches 0, the model relies on calculating the multi-modal prediction result and the cross-entropy loss of the real label to train learning.

[0018] In step two, for each time window, two views are generated by random masking and ,the attention coefficients of the time windows and are calculated as follows:

[0019] ;

[0020] wherein, is a learnable weight matrix, is an attention vector, denotes a feature concatenation operation, the attention coefficient is processed by random masking, finally, the node feature is updated by weighted aggregation:

[0021] ;

[0022] wherein, is a learnable weight matrix, is an attention vector.

[0023] The technical concept of the present application is to divide the time series of pre-processed fMRI data into multiple time windows by using a sliding time window, construct a functional brain network for each window, capture time information across time windows by using multiple spatial attention contrast networks, and obtain a dynamic functional brain network; cross-modal graph neural networks and cross-modal knowledge distillation are used to realize complementary information transmission and modal fusion between modalities, and improve the classification accuracy of depression.

[0024] The present application has the beneficial effects that: by using a multi-modal brain network, a depression image classification method based on spatio-temporal contrast learning and multi-modal brain network fusion is proposed, which is a robust, information-rich and high-precision depression classification method, improves the classification accuracy of depression, and effectively identifies important brain areas related to depression. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1A flowchart of a depression image classification method based on a multi-modal dynamic graph convolutional neural network. DETAILED DESCRIPTION

[0026] The application will be further described below with reference to the drawings.

[0027] Reference Figure 1 A depression image classification method based on a multi-modal dynamic graph convolutional neural network, comprising the following steps:

[0028] Step one: using a sliding window method to divide the time series of functional magnetic resonance imaging (fMRI) data into multiple non-overlapping time windows, and generating a functional connection matrix between brain regions by calculating the correlation of the time series of each window, so as to quantify the functional connectivity between brain regions; this step converts continuous fMRI signals into dynamic functional connection matrices representing the whole brain functional connection mode in the time window , is the number of brain regions, is the total number of windows;

[0029] Step two: extracting high-order spatio-temporal topological features through a multi-channel spatial attention contrast network: the core of the network is a graph attention mechanism (GAT), which aims to adaptively aggregate the information of surrounding nodes; for each time window, two views and are generated by random masking and The attention coefficients are calculated as follows:

[0030] ;

[0031] wherein, is a learnable weight matrix, is an attention vector, denotes a feature concatenation operation. The attention coefficients are processed by random masking, and finally the node features are updated by weighted aggregation:

[0032] ;

[0033] wherein, is a learnable weight matrix, is an attention vector; by aggregating dynamic brain function networks under different views, not only the topological structure of the brain network is preserved, but also the importance between nodes is dynamically adjusted through the attention mechanism.

[0034] Step three, to further enhance the discriminative ability of features, we introduce a contrastive learning constraint, intra-window contrastive loss By maximizing the similarity of features in the same time window across different enhanced views while minimizing the similarity between different brain regions, the intra-window contrastive loss is defined as follows:

[0035] ;

[0036] where, is the cosine similarity between embedding vectors of different views and corresponding to the same time window, is the cosine similarity between embedding vectors of different brain regions in the same view, is a constant; then use the inter-window contrastive loss to further capture the spatio-temporal dependency across time windows, for the same brain region in different windows and , the inter-window contrastive loss constraint is as follows:

[0037] ;

[0038] where, and are the embedding nodes corresponding to the brain region in different time windows, these contrastive constraints enable the model to learn spatio-temporal features with discriminative significance, and the fusion of spatio-temporal features is achieved through self-attention mechanism and LSTM; the embedding features of each window are mapped to Query, Key and Value matrices, and finally the high-order dynamic functional network DBNs is calculated through scaled dot-product attention:

[0039] ;

[0040] Step four, independently encode the input DBNs and structural brain networks to generate modality-specific embedding features, for the DBNs corresponding features and the structural network corresponding features , the cross-modal mapping matrix of the model is calculated as follows:

[0041] ;

[0042] In the cross-modal mapping matrix, the element quantifies the association strength between the DBNs node and the structural node . To ensure that the mapping relationship between DBNs and structural connections conforms to the probability distribution characteristics, then The normalization process is performed to make it a doubly stochastic matrix (the sum of each row and each column is 1), and finally the cross-modal correlation weight matrix is obtained The normalized matrix is used for cross-modal feature mapping, which reflects the weight distribution when cross-modal fusion, based on and The calculation process of the mapping feature of the corresponding mode to another mode is as follows:

[0043] ;

[0044] Step five, aggregate the DBNs and the structural modal features on the dynamic graph structure through bilateral graph convolution, and the normalized is used as the dynamic adjacency matrix to pass cross-modal information. For the first layer convolution, the update formula of the modal is as follows:

[0045] ;

[0046] wherein is a learnable matrix, is an activation function. Through the multi-layer stacked graph convolution, the model gradually aggregates the multi-hop neighbor information, captures the local and global dependency relationship of the functional and structural network, and the multi-modal joint feature is defined as the splicing of the top layer features and of the DBNs modal and the structural modal:

[0047] ;

[0048] Step six, use auxiliary classifiers for the three modes respectively, the auxiliary classifier of the DBNs mode receives the feature after graph convolution, the auxiliary classifier of the structural mode receives the feature after graph convolution, and the auxiliary classifier of the multi-modal joint feature receives the feature after graph convolution, the three modes respectively output single-modal prediction probability distribution , and , and the classifier is usually composed of a fully connected layer and a SoftMax function:

[0049] ;

[0050] Then the consistency of the multi-modal prediction and the single-modal prediction is constrained through KL divergence. Specifically, first, force the multi-modal joint prediction to approximate the prediction of the DBNs mode and the structural mode:

[0051] ;

[0052] ;

[0053] The prediction similarity of DBNs modal and structure modal is further constrained by cross-modal alignment:

[0054] ;

[0055] The total distillation loss is:

[0056] ;

[0057] To reduce the randomness of single-modal prediction results at the beginning of training, the model adopts gradual weight adjustment. Specifically, the weight of distillation loss increases linearly with the training round , , , , , , , ,

[0058] , ;

[0059] , , , , ,

[0060] , , ,

[0061] , , ,

[0062] , .

[0063] The implementation process of the embodiment is as follows: the data set used in the application is a data set of Guangzhou Medical University, and the multi-modal magnetic resonance imaging data and demographic statistical data of all subjects are collected in the Radiology Department of the Affiliated Brain Hospital of Guangzhou Medical University; all subjects include basic demographic data, Hamilton depression scale scores and three modal magnetic resonance imaging data: structural magnetic resonance imaging data, resting state functional magnetic resonance imaging data and diffusion tensor imaging data; the five-fold cross-validation method is used to divide the data set into a training set, a test set and a validation set, and the whole sample is randomly divided into 5 non-repeated subsets, of which 3 subsets are used as a training set for model parameter learning, and the remaining 2 subsets are used for model hyperparameter tuning and test set respectively; the application trains the model in the training set, adjusts the model hyperparameters in the validation set through two evaluation indexes of classification accuracy and F1 score, and finally inputs the multi-window functional connectivity matrix and structural brain network of each subject in the test set, so as to assist in judging whether each subject is ill, and the classification accuracy and F1 score of the test set are calculated.

[0064] In order to evaluate the classification performance of the model, the proposed method is compared with common machine learning methods, deep learning and multi-modal feature fusion methods that have achieved good performance in other studies. Machine learning methods include support vector machine (SVM) and multilayer perceptron (MLP), deep learning method is BrainGNN, and the compared multi-modal feature fusion method is M-GCN. In a multi-modal self-sampling depression data set, the method is better than other methods, the classification accuracy is 80.32%, and the F1 score is 92.86%.

[0065] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and is only for the purpose of description. The protection scope of the application should not be regarded as being limited to the specific forms described in the embodiments, and the protection scope of the application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A depression image classification method based on a multi-modal dynamic graph convolutional neural network, characterized in that, Comprising the following steps: Step one, using sliding window method to divide the time series of functional magnetic resonance imaging data into multiple non-overlapping time windows, each window generates functional connectivity matrix by calculating the correlation between the time series of brain regions representing the whole brain functional connectivity pattern within the time window , is the number of brain regions, is the total number of windows, thus quantifying the functional connectivity between brain regions; Step two, extract high-order spatio-temporal topology features through multi-channel spatial attention contrast network; For each time window, two views g 1 (t) and g 2 (t) are generated by random masking and attention coefficients of brain regions are computed as follows: ； wherein, is a learnable weight matrix, is an attention vector, denotes a feature concatenation operation, the attention coefficients are processed by a random mask, and finally, the node features are updated by weighted aggregation: ； wherein, is a learnable weight matrix; Step three, introduce contrast learning constraints, the fusion of spatio-temporal features is realized through self-attention mechanism and LSTM, and high-order dynamic functional network DBNs are calculated through scaling dot product attention; Step four, independently encode the input DBNs and structural brain networks to generate modality-specific embedding features; Step five, aggregate DBNs and structural modality features on dynamic graph structure through bilateral graph convolution; Step six, use auxiliary classifiers for DBNs modality, structural modality and multi-modal joint features respectively, output single modality prediction probability distribution and multi-modal prediction probability distribution respectively, and then constrain the consistency of multi-modal prediction and single modality prediction through KL divergence; The model uses gradual weight adjustment. The auxiliary classifier of the DBNs modal receives the graph-convolved features in step six The auxiliary classifier of the structure modal receives the graph-convolved features The auxiliary classifier of the multi-modal joint features receives the graph-convolved features The three modals respectively output single-modal prediction probability distribution , and multi-modal prediction probability distribution The auxiliary classifier is composed of a fully connected layer and a SoftMax function.

2. The depression image classification method based on the multi-modal dynamic graph convolutional neural network according to claim 1, wherein, In the step three, the intra-window contrastive loss By maximizing the feature similarity in different enhanced views for the same time window, while minimizing the similarity between different brain regions; these contrastive constraints enable the model to learn discriminative spatio-temporal features; The embedding vectors of each window are mapped as Query, Key and Value matrices. The embedding vectors of each window are mapped as Query, Key and Value matrices.

3. The depression image classification method based on the multi-modal dynamic graph convolutional neural network according to claim 1 or 2, characterized in that, In step four, the corresponding features of DBNs and structural brain networks are calculated, and the cross-modal mapping matrix of the model is calculated ; then is normalized to become a double random matrix, and finally the cross-modal correlation weight matrix is obtained, and the normalized matrix is used for cross-modal feature mapping, which reflects the weight distribution when cross-modal fusion is performed. 4.The depression image classification method based on the multi-modal dynamic graph convolutional neural network according to claim 1, wherein, In step six, the model employs a gradual weight adjustment process as follows: weighting the distillation loss. With training rounds Linear growth, in the early stages of training , Approaching 0, the model depends on calculating multimodal prediction results. With real labels Cross-entropy loss Come for training and learning.

Images