Automatic identification method of mild cognitive impairment based on heterogeneous graph neural network

Abstract

This invention relates to an automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks. The method includes the following steps: S1, acquiring resting-state functional magnetic resonance imaging (fMRI) images and diffusion tensor imaging (DTI) images, constructing a heterogeneous graph based on the images, and calculating node features; S2, constructing adjacency matrices for four types of meta-paths based on structure-function coupling common community search, obtaining the adjacency matrix and node feature matrix of the heterogeneous graph; S3, reconstructing the meta-paths to form a new heterogeneous graph; S4, inputting the heterogeneous graph, the new heterogeneous graph, and their corresponding adjacency matrix and node feature matrix into a mild cognitive impairment identification model, and the mild cognitive impairment identification model outputs the identification result. Compared with the prior art, this invention has the advantages of fully considering the heterogeneity of the heterogeneous graph in the neural network and improving the classification performance of the mild cognitive impairment identification model.

Description

Technical Field

[0001] This invention relates to the field of computer-aided diagnostic technology, and in particular to an automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks. Background Technology

[0002] Mild cognitive impairment (MCI) is a transitional state between normal aging and Alzheimer's disease (AD), characterized by mild but significant decline in memory or other cognitive functions. Although individuals with MCI have relatively intact daily living abilities, they have a higher risk of developing Alzheimer's disease or other forms of dementia; therefore, MCI is considered a high-risk stage for AD progression. Currently, MCI can be further divided into early mild cognitive impairment (EMCI) and late mild cognitive impairment (LMCI).

[0003] Objective and automated identification of MCI is of great significance. On the one hand, early identification of MCI can provide patients with timely intervention opportunities and may delay further deterioration of cognitive function; on the other hand, automated identification can reduce reliance on traditional subjective assessment methods and reduce subjective bias in diagnosis. Previous studies have shown that resting-state functional magnetic resonance imaging (rs-fMRI) and diffusion tensor imaging (DTI) have made important contributions to understanding brain structure and function (references [1], [2]). Specifically, functional connectivity (FC) of the brain constructed by rs-fMRI imaging can capture spontaneous neuronal activity and reveal the intrinsic connections between different brain regions (references [3], [4]); while structural connectivity (SC) of the brain constructed by DTI imaging can provide important insights into the integrity of white matter structure and the identification of neurofibrillary abnormalities (references [5], [6]). In recent years, many studies have found that changes in neurological function are directly related to changes in white matter structure (references [7], [8]). Therefore, by integrating multimodal neuroimaging data such as rs-fMRI and DTI, and using deep learning models for feature extraction and analysis, the accuracy and efficiency of MCI identification can be effectively improved. This automatic identification method based on objective data not only provides an important auxiliary tool for clinical diagnosis but also provides a scientific basis for studying the pathogenesis of MCI to AD.

[0004] Since FC and Structural Connections (SC) can be easily described using graph structure data, the framework based on graph neural networks (GNNs) has become a popular choice for identifying mild cognitive impairment (MCI) by combining bimodal information (references [9] to

[12] ). Currently, there are two main popular ways to use GNNs to fuse bimodal information: feature-level fusion and edge-level fusion. Specifically, in feature-level fusion, the same backbone network is used to extract functional and structural features from different modalities, and then these features are concatenated or weighted and summed as the fused features for further analysis. In edge-level fusion, functional connections (FC) and structural connections (SC) are usually integrated into an isomorphic graph that summarizes structural and functional connection information, and then the fused structural-functional features are extracted from this summary graph. Although these methods can fuse bimodal features, there are still some shortcomings. First, the feature interaction of the feature-level fusion method only occurs in the concatenation or weighted summation stage, resulting in insufficient feature fusion and a decrease in classification performance. Although homogeneous summation graphs in edge-level fusion provide more consistent feature embeddings, graphs constructed in a homogeneous manner may disrupt the inherent heterogeneity between FC and SC, such as differences in feature space and topology. Summary of the Invention

[0005] The purpose of this invention is to provide an automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks, in order to fully consider the heterogeneity of heterogeneous graphs in neural networks and improve the classification performance of mild cognitive impairment identification models.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] An automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks, the method comprising the following steps:

[0008] S1. Acquire resting-state functional magnetic resonance imaging and diffusion tensor imaging, and construct heterogeneous maps based on the images. The heterogeneous graph includes two different types of nodes: functional brain region type nodes. and structural brain region type nodes The heterogeneous graph includes four types of meta-paths, namely the first meta-path. Secondary path Third Path Fourth Path Wherein, Φ1 and Φ2 are isomorphic paths, while Φ3 and Φ4 are heteromorphic paths; calculate the node features corresponding to functional brain region nodes and structural brain region nodes respectively;

[0009] S2. Based on the structure-function coupling common community search, the adjacency matrix of four meta-paths is constructed. Based on the adjacency matrix of the four meta-paths, the adjacency matrix of the heterogeneous graph is obtained. The node feature matrix of the heterogeneous graph is obtained by concatenating the node feature matrix of the functional brain region type and the node feature matrix of the structural brain region type.

[0010] S3. Reconstruct the first metapath Φ1 using a dynamic sliding window method to obtain the metapath. Simultaneously, the reconstructed metapath is obtained. and Update metapath and The adjacency matrix forms a new heterogeneous graph. The adjacency matrix, the node feature matrix of the new heterogeneous graph, and the heterogeneous graph before reconstruction. The node feature matrix remains consistent;

[0011] S4, Heterogeneous Diagram and new heterogeneous graphs The adjacency matrix and node feature matrix are input together into the mild cognitive impairment identification model. The mild cognitive impairment identification model is constructed based on the heterogeneous graph attention neural network (HAN), which includes an improved heterogeneous graph attention pooling layer. The mild cognitive impairment identification model outputs the identification result.

[0012] Furthermore, the adjacency matrix of the first path is:

[0013]

[0014] in, This represents the i-th row and j-th column of the adjacency matrix of the first-order path. E represents the node feature of node i in the functional brain region type node, d represents the dimension of the corresponding node feature, and E i σ represents the mean of the features of node i. i Cov represents the variance of the features of node i. ij This represents the covariance between node i and node j in the functional brain region type nodes.

[0015] Furthermore, the calculation process of the adjacency matrix of the second-order path is as follows: deterministic white matter fiber tract tracking is performed between every two regions of interest (ROIs), and the number of tracked white matter fiber tracts is used as the edge weight between the nodes of these two structural brain regions, which is the i-th row and j-th column of the adjacency matrix of the second-order path. When the edge weight is less than the threshold, it is discarded. The structural brain region type node is a region of interest (ROI) divided by diffusion tensor imaging.

[0016] Furthermore, the calculation process for the adjacency matrix of the third-element path and the adjacency matrix of the fourth-element path is as follows:

[0017] For a pair of cross-modal nodes Their connection modes in their respective modalities are as follows: and Calculate the cosine similarity ε between these two connection modes. ij For each node i of a functional brain region type node and a node of a structural brain region type node, the cosine similarity between their connection patterns is calculated to obtain a vector. In vector The values ​​of the 8 most similar connection patterns are selected and used as the i-th row of the first adjacency matrix. The value at the corresponding position in the middle;

[0018] A common community search based on structure-function coupling is adopted to find a closed subgraph g consisting of 3 nodes. The closed subgraph g satisfies the condition that it exists in both Φ1 and Φ2. The second adjacency matrix is ​​calculated based on the closed subgraph g.

[0019] Add the first adjacency matrix and the second adjacency matrix to obtain the adjacency matrix of the third path;

[0020] Transpose the adjacency matrix of the third-order path to obtain the adjacency matrix of the fourth-order path.

[0021] Furthermore, the first adjacency matrix is:

[0022]

[0023] in, Represents the i-th row and j-th column of the first adjacency matrix, TopK j This indicates selecting the indices of the K largest elements in the vector; here, K is set to 8.

[0024] Furthermore, the second adjacency matrix is:

[0025]

[0026] in, It represents the i-th row and j-th column of the second adjacency matrix.

[0027] Furthermore, the specific steps of S3 are as follows:

[0028] A sliding window method is used to perform a sliding window operation on the node features corresponding to nodes in functional brain regions. Within each window, multiple local meta-paths are constructed using the Pearson correlation coefficient, considering the inclusion of edges. Three-node subgraph g ij ,side This represents an edge between two functional brain region type nodes. Weights are assigned to this edge in three different types of subgraphs corresponding to the three possible edge scenarios in the subgraph. Count the three types of subgraphs g containing this edge. ij The total number of occurrences in the local metapath is denoted by a vector. The three edge connection cases are: containing only edges Includes edges And any other edge, and containing The closed subgraph formed by the other two edges;

[0029] The metapath obtained by computational reconstruction adjacency matrix Keeping the adjacency matrix of the second path Φ2 unchanged, based on the adjacency matrix Update metapath using the S2 method and The adjacency matrix forms a new heterogeneous graph. and new heterogeneous graphs First path Secondary path Φ2, tertiary path Fourth Path and

[0030] Furthermore, the adjacency matrix for:

[0031]

[0032] in, Indicates the presence of edges Three types of three-node subgraphs g ij The total number of occurrences in the local metapath This represents the transpose of the weights.

[0033] Furthermore, the heterogeneous graph attention neural network HAN includes 3 heterogeneous graph attention convolutional layers, 3 improved heterogeneous graph attention pooling layers, and 3 graph readout layers.

[0034] Furthermore, the computational flow within the improved heterogeneous graph attention pooling layer is as follows:

[0035] For the heterogeneous graph output of the l-th layer heterogeneous graph attention convolutional layer of the network, the attention score matrix of each node type is calculated, and then the pooling matrix is ​​calculated. The pooling matrix is ​​applied to the node feature matrix and adjacency matrix of the heterogeneous graph. The result of the pooling is used as the output of the improved heterogeneous graph attention pooling layer.

[0036] The attention score matrix is ​​as follows:

[0037]

[0038] Among them, S (l) Represents the attention score matrix. θ (l) These represent the input adjacency matrix, input node feature matrix, and parameter matrix of the current layer received by the l-th layer of the neural network, respectively. The subscript f corresponds to the functional state, and the subscript d corresponds to the structural state.

[0039] The pooling matrix is:

[0040]

[0041] Among them, P (l) D represents the final pooling matrix after unifying the dimensions using an all-zero matrix. (l) Let O represent the dense pooling information matrix to be unified in terms of dimension, and let b represent the matrix of all zeros. (l) W represents the bias vector in the linear layer. (l) The parameter matrix represents the linear layer, with the subscript f corresponding to the functional state and the subscript d corresponding to the structural state.

[0042] The result after pooling is:

[0043]

[0044] in, This represents the result after pooling. These represent the adjacency matrices corresponding to the first, third, and second ary paths in the heterogeneous graph of the input neural network, respectively.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) When constructing metapaths within the same node type, this invention adopts a construction method of functional connectivity (FC) and structural connectivity (SC) to capture and preserve brain function and structural information reflected in neuroimaging to the greatest extent. When constructing metapaths between different node types, this invention designs two different methods based on structural-functional coupling and common community search to fully reflect cross-modal node interaction information and improve the classification performance of the mild cognitive impairment identification model.

[0047] (2) The present invention improves the pooling strategy of the heterogeneous graph attention pooling layer, makes full use of heterogeneous information, and prevents feature confusion during pooling, thereby further improving the classification performance of the mild cognitive impairment recognition model.

[0048] (3) By constructing a new heterogeneous graph, the present invention can effectively enhance a minority of samples and alleviate the problem of data sample imbalance. Attached Figure Description

[0049] Figure 1 A detailed flowchart of the method proposed in this invention;

[0050] Figure 2 A block diagram for constructing heterogeneous element paths in a heterogeneous graph;

[0051] Figure 3 This is a comparison chart of experimental results, in which... Figure 3 (a) is the t-SNE dimensionality reduction visualization of the high-dimensional features extracted by the method proposed in this invention in the comparison experiment between MCI and NC. Figure 3 (b) shows the t-SNE dimensionality reduction visualization of the high-dimensional features extracted by the method proposed in this invention in the comparison experiment between EMCI and NC. Figure 3 In the (c) LMCI and NC comparison experiment, the high-dimensional features extracted by the method proposed in this invention are visualized by t-SNE dimensionality reduction.

[0052] Figure 4 This is a schematic diagram of the top 10 brain regions most relied upon in distinguishing different types of mild cognitive impairment using both resting-state functional magnetic resonance imaging (fMRI) and diffusion tensor imaging modalities, as described in this invention. Figure 4 (a) shows the top 10 brain regions most relied upon by the neural network proposed in this invention in distinguishing different mild cognitive impairments in the resting-state functional magnetic resonance imaging modality. Figure 4 (b) shows the top 10 brain regions most relied upon by the neural network proposed in this invention in the diffusion tensor imaging modality when distinguishing different mild cognitive impairments;

[0053] Figure 5 ROC curves for different data augmentation ratios. Detailed Implementation

[0054] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0055] This invention proposes an automated identification method for mild cognitive impairment (MCI) by integrating functional and structural connectivity in the brain using a heterogeneous graph neural network. First, using oxygenation-dependent and white matter fiber tract information provided by resting-state functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI), the invention divides the brain into 90 functional nodes and 90 structural nodes, using 187-dimensional time-series information and 57-dimensional radiomics features as node features for the functional and structural nodes, respectively. Further, based on the correlation between time-series information and the number of white matter fiber tract connections across brain regions, the invention establishes meta-pathways within isomorphic modalities; and establishes cross-modal meta-pathways based on structure-function coupling and cross-modal brain region connectivity similarity. Finally, a heterogeneous graph containing 180 nodes and four different meta-pathways is constructed to describe the brain of each patient. Ultimately, the heterogeneous graph of each case is processed by a heterogeneous graph neural network to extract 256-dimensional features, which are then classified using a multilayer perceptron. This invention, as a novel method based on heterogeneous graph neural networks, provides a new automated assessment scheme for the diagnosis and identification of mild CMI. A detailed flowchart of the method is shown below. Figure 1 As shown, it includes the definition and feature extraction of nodes in heterogeneous graphs, the construction of meta-paths and their adjacency matrices in heterogeneous graphs, data augmentation based on heterogeneous graphs, feature extraction based on neural networks, and classification decision-making.

[0056] The specific steps of this invention are as follows:

[0057] S1. Acquire resting-state functional magnetic resonance imaging and diffusion tensor imaging, and construct heterogeneous maps based on the images. The heterogeneous graph includes two different types of nodes: functional brain region type nodes. and structural brain region type nodes The heterogeneous graph includes four types of meta-paths, namely the first meta-path. Secondary path Third Path Fourth Path Wherein, Φ1 and Φ2 are isomorphic paths, while Φ3 and Φ4 are heteromorphic paths; calculate the node features corresponding to functional brain region nodes and structural brain region nodes respectively;

[0058] S2. Based on the structure-function coupling common community search, the adjacency matrix of four meta-paths is constructed. Based on the adjacency matrix of the four meta-paths, the adjacency matrix of the heterogeneous graph is obtained. The node feature matrix of the heterogeneous graph is obtained by concatenating the node feature matrix of the functional brain region type and the node feature matrix of the structural brain region type.

[0059] S3. Reconstruct the first metapath Φ1 using a dynamic sliding window method to obtain the metapath. Simultaneously, the reconstructed metapath is obtained. and Update metapath and The adjacency matrix forms a new heterogeneous graph. The adjacency matrix of the new heterogeneous graph remains consistent with the node feature matrix before reconstruction.

[0060] S4, Heterogeneous Diagram and new heterogeneous graphs The adjacency matrix and node feature matrix are input together into the mild cognitive impairment identification model. The mild cognitive impairment identification model is constructed based on the heterogeneous graph attention neural network (HAN), which includes an improved heterogeneous graph attention pooling layer. The mild cognitive impairment identification model outputs the identification result.

[0061] In S1, the heterogeneous graph defines two different types of nodes: functional and structural brain region nodes. Their physical meanings both correspond to actual brain regions, but because their node features have different semantics—that is, they describe brain regions from functional and structural semantic perspectives respectively—they are considered different node types.

[0062] Graph construction is a crucial step in the identification of cognitive impairment. With increasing research focusing on combining resting-state functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI) information, the rational construction of graphs that facilitate the fusion of bimodal image information is becoming increasingly important. This invention proposes constructing heterogeneous graphs to fuse bimodal image information. Two different types of nodes are defined in the heterogeneous graph: functional and structural brain region nodes, denoted as […]. and Their physical meanings all correspond to actual brain regions, but because their node features have different semantics—that is, they describe brain regions from functional semantics and structural semantics respectively—they are considered different node types.

[0063] For functional node segmentation and feature extraction, the resting-state functional magnetic resonance imaging (fMRI) images were preprocessed using the DPABI tool (reference

[15] ): Since the magnetic field in the first 10 time points of the resting-state fMRI was unstable, it was discarded as temporal noise; after head motion correction, the images were registered to the MN152 standard brain atlas and filtered using a bandpass filter with a bandwidth of 0.01-0.1 Hz; the brain was divided into 90 regions of interest (ROIs) using the Automatic Anatomical Labelling (AAL) template, and each ROI was treated as a functional node; the average grayscale value was calculated for each ROI at each time point, and the 187-dimensional average time series was used as the node feature of the functional node. Specifically, after head motion correction, the images were registered to the MN152 (Montreal Neurological Institute (MNI)) standard brain atlas, and the node features were denoted as...

[0064] For structural node segmentation and feature extraction, the PANDA toolkit (reference

[16] ) was used to preprocess the diffusion tensor imaging images: the diffusion index was calculated to generate the fractional anisotropy FA map; it was registered to the MN152 standard brain map set, and the brain region automatic anatomical labeling brain region template AAL was used to divide the brain space in the FA map into 90 regions of interest (ROIs), and each ROI was used as a structural node; 57 radiomics features such as grayscale and texture were calculated for each ROI region in the FA map, which were used as the node features of the structural node.

[0065] In S2, four meta-paths are defined in the heterogeneous graph, namely: Among them, Φ1 and Φ2 are isomorphic paths, while Φ3 and Φ4 are heteromorphic paths, and Φ4 can be regarded as a flip of Φ3.

[0066] For isomorphic path Φ1, its adjacency matrix is ​​constructed using the Pearson correlation coefficients between node features. Considering that an absolute value of the Pearson correlation coefficient less than 0.4 indicates a weak or no correlation between the two variables, all values ​​less than 0.4 are discarded from the adjacency matrix. For isomorphic path Φ2, its adjacency matrix is ​​constructed using the following steps: Deterministic white matter fiber tract tracking is performed between every two ROIs using the PANDA toolkit, and the number of tracked white matter fiber tracts is used as the edge weight between these two nodes. To ensure the effectiveness and sparsity of the adjacency matrix, all values ​​less than 5 are discarded.

[0067] For isomorphic path Φ1, its corresponding adjacency matrix A f Constructed by the following steps: Record each The node characteristics of type i are The edge weights between nodes i and j are the Pearson correlation coefficients of their node features, and the formula is as follows:

[0068]

[0069] Considering that an absolute value of the Pearson correlation coefficient less than 0.4 indicates a weak or no correlation between the two variables, all values ​​less than 0.4 are discarded in the adjacency matrix.

[0070] For isomorphic path Φ2, its corresponding adjacency matrix A d The process is constructed using the following steps: Utilizing the PANDA toolkit, deterministic white matter fiber bundle tracking is performed between every two ROIs. The number of tracked white matter fiber bundles is used as the weight of the edge between these two nodes. The value of . To ensure the effectiveness and sparsity of the adjacency matrix, all values ​​less than 5 in the adjacency matrix are ultimately discarded.

[0071] For the heterogeneous primitive path Φ3, its adjacency matrix is ​​constructed using the following steps: Measuring the similarity of connection patterns between cross-modal node pairs. For each cross-modal node pair, calculating the Pearson correlation coefficient of its connection patterns in its respective modality. Finally, for each functional node, selecting the top 8 structural nodes with the most similar connection patterns and establishing edges between them; Community search based on structure-function coupling, i.e., searching for closed subgraphs consisting of 3 nodes that exist simultaneously in Φ1 and Φ2, and then adding the edges existing in these closed subgraphs to the adjacency matrix, with edges starting from functional nodes and terminating at structural nodes. Finally, for the heterogeneous primitive path Φ4, its corresponding adjacency matrix is ​​the transpose of the adjacency matrix of Φ3.

[0072] The construction of heterogeneous primitive path Φ3 is as follows Figure 2 As shown, its corresponding adjacency matrix A fd It is constructed by the following steps: On the one hand, measuring the similarity of connection patterns of cross-modal node pairs: for a pair of cross-modal node pairs Their connection modes in their respective modalities are as follows: and Calculate the cosine similarity ε of their connection patterns. ij For each Node i of type i, which can be associated with Calculate the connection pattern similarity for each of the 90 nodes in the type, and express it as a vector. This indicates. Ultimately, in The values ​​of the top 8 most similar connection patterns are selected as the values ​​of the corresponding positions in the adjacency matrix from this perspective. The formula is as follows:

[0073]

[0074] On the other hand, a community search based on structure-function coupling is adopted. That is, for a closed subgraph g consisting of 3 nodes, if it exists in both Φ1 and Φ2, the three edges of the closed subgraph are added to the adjacency matrix A. fd In the middle, it is believed that its first node comes from Type, the second node comes from Type, formula as follows:

[0075]

[0076] Finally, by summing the results of the two steps, the adjacency matrix A can be constructed. fd :

[0077]

[0078] Finally, for the heterogeneous primitive path Φ4, its corresponding adjacency matrix A df That is, A fd transpose Through the above two steps, the final heterogeneous graph is constructed. The node feature matrix X H With A H It can be calculated using the following formula:

[0079]

[0080]

[0081] In S3, the rich heterogeneity of the heterogeneous graph allows for the reconstruction of meta-paths from different perspectives, thereby achieving data augmentation while preserving semantics. Considering that the edge relationships between functional nodes described by Φ1 can be reconstructed using a dynamic sliding window approach... Without affecting its functional semantics, and on the other hand, Φ3 and Φ4 are also directly affected by Φ1 during the construction process, so we can refactor them. This allows us to reconstruct the three metapaths in the heterogeneous graph. On the other hand, since Φ2 actually strictly reflects the physical structural information between brain regions, Φ2 must be preserved to ensure the stability of brain structural information.

[0082] The corresponding adjacency matrix is ​​constructed using the following steps: A sliding window method is used, with a window length of 90 and a step size of 1, across 187-dimensional time series information. Within each window, 97 local meta-paths are constructed using the Pearson correlation coefficient. Considering a three-node subgraph containing a specific edge, weights of 0.01, 0.02, and 0.1 are assigned based on the possible edges within the subgraph. The total frequency of occurrences of subgraphs containing different edges in the 97 local meta-paths is counted, and the weighted average of these frequencies is used as the weight value for that edge. Finally... and Its adjacency matrix can be constructed according to step two and automatically reflect... The changes introduced. The specific steps are as follows:

[0083] 1) Using the sliding window method, with a window length of 90 and a step size of 1, a sliding window was applied to the 187-dimensional time series information. Within each window, a total of 97 local meta-paths were constructed using the Pearson correlation coefficient. 2) Consider including edges Three-node subgraph g ij , specific edge The selection is achieved through traversal, meaning that for each edge in the new adjacency matrix, it is assumed that it may exist, and then the edges are selected iteratively. Based on the possible edge combinations in the subgraph, it is categorized into three types: 1-edge, 2-edge, and closed subgraph (containing only the edge of interest; the edge of interest plus any other edge); and a subgraph containing all three edges, abbreviated as g. ij 3) Statistics include edge e ij subgraph g ij In 97 The total number of occurrences is denoted as a vector. For different g ij edge e in ij Assign weight vector The weight vector assigns weights to these three different types of subgraphs, and its dimension is 1*3. The three different types of subgraphs are counted in 97... The number of times something appears in a vector is used to obtain a 1*3 dimension vector. Finally, the following formula is used to calculate: The value at the corresponding position:

[0084]

[0085] Next, and Its adjacency matrix can be constructed according to formulas (5)-(7), and the final data-enhanced heterogeneous graph is obtained. Node feature matrix and It can be calculated by formulas (8)-(9). In S4, the heterogeneous graph attention neural network HAN originates from the improvement of the traditional graph neural network GNN. By introducing node-level and semantic-level attention, it realizes attention operations within the same node type and on different meta-paths, fully considering the heterogeneity of the heterogeneous graph in the neural network.

[0086] A feature extraction method based on GNN was adopted. The GNN network includes three heterogeneous graph attention convolutional layers, three heterogeneous graph attention pooling layers, and three graph readout layers. The following innovative improvements and adjustments were made: First, a heterogeneous graph attention pooling layer was introduced to score the attention of each type of node and filter out nodes with low scores. This reduces the model complexity while ensuring that the model focuses on the core brain regions, thereby improving the model's discrimination accuracy and clinical interpretability. Second, a graph readout layer combining max average pooling and max pooling was designed to ensure that more important information in the graph is retained during each graph readout layer.

[0087] The HAN-based feature extraction method designed in this invention has the following characteristics: Figure 2The structure of the network includes three heterogeneous graph attention convolutional layers, three heterogeneous graph attention pooling layers, and three graph readout layers. The heterogeneous graph attention pooling network in reference

[17] is designed for social networks. Since the information in the graphs of medical neuroimaging is more complex, and it is important to distinguish and consider nodes and their connections under functional semantics and structural semantics, the following adjustments were made to the pooling network: a single-layer heterogeneous graph attention network layer is used to calculate the attention score matrix for each type of node, and the attention score matrix is ​​further used as the pooling matrix to perform matrix dot product with the original adjacency matrix to realize the pooling function. While reducing the complexity of the model, it ensures that the model pays attention to the core brain regions, thereby improving the model's discrimination accuracy and clinical interpretability.

[0088] This invention sets the number of attention heads in the heterogeneous graph attention convolutional layer to 8 to accelerate the calculation speed of attention; sets the network hidden layer dimension to 128, the pooling score of the heterogeneous graph attention pooling layer to 0.8, the dropout score to 0.45, and the batch input data size to 32; uses the AdamW optimizer for backpropagation to update network parameters, and sets the initial learning rate to 0.0001, the initial weight decay coefficient to 0.0001, and the maximum number of updates during network training to 200 to ensure sufficient network convergence; additionally, a pairwise normalization strategy is introduced during network training to reduce the oversmoothing problem, and a cosine annealing strategy is introduced to accelerate network convergence.

[0089] The adjusted heterogeneous graph attention pooling layer is implemented as follows: For the heterogeneous graph output by the l-th convolutional layer of the network, an attention score matrix is ​​calculated for each node type using a HAN layer.

[0090]

[0091] Further analysis of the fractional matrix S (l) Perform densification and zero-matrix filling operations, focusing on specific node types, to ensure that information from different node types is not mixed up during subsequent pooling:

[0092]

[0093] Finally, the pooling matrix P is... (l) The pooling result is calculated by applying the node feature matrix and adjacency matrix:

[0094]

[0095] Finally, classification is performed by inputting the 256-dimensional high-dimensional features extracted from the neural network in step four into a multi-head perceptron (MLP) network consisting of three fully connected layers and activation functions for binary classification. Five-fold cross-validation is used as the validation model, and four quantitative metrics are used to evaluate the recognition performance: accuracy, sensitivity, specificity, and area under the ROC curve.

[0096] This invention employs 5-fold cross-validation as the validation model. To comprehensively evaluate the method's performance, four metrics are used for quantitative assessment: accuracy (ACC), sensitivity (SEN), specificity (SPE), and area under the ROC curve (AUC). The calculation methods for each metric are as follows:

[0097]

[0098] TP, FP, TN, and FN represent the number of correct positives, incorrect positives, correct negatives, and incorrect negatives, respectively.

[0099] The system corresponding to this invention comprises five modules: a node definition and node feature extraction module, a meta-path and its adjacency matrix construction module, a data augmentation module, a feature extraction module, and a classification decision module. These modules are respectively used to perform the five steps of the recognition method: defining and extracting node features from heterogeneous graphs, constructing meta-paths and their adjacency matrices in heterogeneous graphs, data augmentation based on heterogeneous graphs, feature extraction based on neural networks, and classification decision. Specifically, the node definition and node feature extraction module and the meta-path and its adjacency matrix construction module are responsible for constructing a reasonable heterogeneous graph input; the data augmentation module is responsible for generating more training cases to alleviate the problem of imbalanced dataset samples.

[0100] This invention focuses on addressing some challenges in introducing heterogeneous graphs into the fusion analysis of rs-fMRI and DTI bimodal neuroimages. Specifically: 1) Different types of relationships in heterogeneous graphs are usually defined by semantically dependent metapaths (references

[18]

[19] ). This means that when constructing metapaths, they must reflect the inherent relationships between nodes within a node type and reveal the interactive relationships between nodes of different node types. In this regard, this invention adopts the construction methods of FC and SC when constructing metapaths within the same node type to capture and preserve the brain functional and structural information reflected in neuroimages to the greatest extent. When constructing metapaths between different node types, this invention designs two different methods based on structure-function coupling and common community search to fully reflect cross-modal node interaction information; 2) Pooling strategies for heterogeneous graphs need to deal with more complex relationships. Directly applying pooling strategies designed for homogeneous graphs (references

[20]

[21] ) may lead to feature confusion between different node types in heterogeneous graphs. Therefore, this invention references and improves the pooling strategy in reference

[17] , which can automatically balance the information interaction in different meta-paths, make full use of heterogeneous information, and prevent feature confusion during pooling. 3) The difference in the incidence of different subtypes of mild cognitive impairment leads to data sample imbalance, which affects the classification performance of deep learning networks. Therefore, this invention combines the characteristics of rs-fMRI image analysis and proposes a simple and efficient data augmentation method based on heterogeneous graphs, which can effectively augment a minority of samples and alleviate the problem of data sample imbalance.

[0101] The overall steps of this invention are as follows:

[0102] First, resting-state functional magnetic resonance imaging (fMRI) images underwent temporal correction, skull removal, head motion correction, registration to the MNI standard brain space, and bandwidth filtering. Then, the brain was divided into 90 regions of interest (ROIs) using the Automatic Anatomical Labeling (AAL) template. Each RPI was used as a functional node, and the time series of the average grayscale value within that RPI was used as the node feature.

[0103] Then, the diffusion tensor imaging was resampled, descrambled, diffusion parameters were calculated, registered to the MNI standard brain space, diffusion parameters were smoothed, and white matter fiber tracts were deterministically traced. The brain was then divided into 90 regions of interest (ROIs) using the Automatic Anatomical Labeling (AAL) template. Each RPI was used as a structural node, and 57 radiomics features, including grayscale and texture, extracted from the FA parameter map within that RPI were used as node features.

[0104] Within the set of functional nodes, the Pearson correlation coefficient between node features is calculated to construct brain functional connections as isomorphic paths Φ1. Based on the properties of the Pearson correlation coefficient, edges with absolute weight values ​​less than 0.4 are discarded.

[0105] Within the set of structural nodes, brain structural connections are constructed as isomorphic paths Φ2 based on the number of fiber bundles traced between nodes by deterministic white matter fiber bundles. To ensure the sparsity and efficiency of the graph, edges with fewer than 5 fibers are discarded.

[0106] Between functional and structural nodes, the cosine similarity of node connection patterns is calculated, and cross-modal connections are established between the top 8 node pairs with the most similar connection patterns, constructing the skeleton of heterogeneous metapath Φ3. Simultaneously, in brain functional and structural connectivity, a community search method is used to assign weights of 0.01, 0.02, and 0.1 to an edge in different three-node communities, respectively. The weight values ​​correspond to the number of edges present in the three-node community. Finally, a weighted sum is calculated based on the frequency of community occurrences, serving as a supplement to Φ3 to increase its diversity and connectivity. Φ4 is obtained by flipping Φ3.

[0107] Data augmentation based on heterogeneous graphs was performed, increasing the number of mild cognitive impairment data (100, 48, and 21 cases) to 200, 240, and 210 cases respectively. These data were then combined with a normal control group of 240 cases to generate three cognitive impairment datasets with a total size of 440, 480, and 450 cases, which were used as input datasets for the neural network.

[0108] The two modes (heterogeneous graph) in the heterogeneous graph and new heterogeneous graphs The node feature matrix of each node and the adjacency matrices corresponding to the four meta-paths are input into the deep learning network proposed in this invention. This network consists of three heterogeneous graph attention convolutional layers, three heterogeneous graph attention pooling layers, and three graph readout layers. The number of attention heads in the heterogeneous graph attention convolutional layers is set to 8, the hidden layer dimension is 128, the pooling score of the heterogeneous graph attention pooling layers is 0.8, the dropout score is 0.45, and the batch input data size is 32. The AdamW optimizer is used for backpropagation to update network parameters, with an initial learning rate of 0.0001, an initial weight decay coefficient of 0.0001, and a maximum number of updates during network training set to 200 to ensure sufficient network convergence. Pairwise normalization is introduced during network training to reduce oversmoothing, and cosine annealing is used to accelerate network convergence. Focal loss is used as the loss function to further reduce the impact of imbalanced data samples on training. The graph readout layer consists of a global max pooling layer and a global average pooling layer. The concatenation of the outputs of the two layers serves as the feature vector for the entire graph readout.

[0109] The 256-dimensional high-dimensional features extracted by the neural network are input into a multi-head perceptron (MLP) network consisting of three fully connected layers and activation functions for binary classification. This invention employs 5-fold cross-validation as the validation model. To comprehensively evaluate the method's performance, four metrics—accuracy, sensitivity, specificity, and area under the ROC curve—are used for quantitative evaluation.

[0110] This invention collected data from 409 cases from the Alzheimer's Disease Neuroimaging Project (ADNI) dataset, including 100 cases of mild cognitive impairment (MCI), 48 cases of early mild cognitive impairment (EMCI), 21 cases of late mild functional cognitive impairment (LCI), and 240 normal controls (NC). Each case data simultaneously included bimodal neuroimaging data from resting-state functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI).

[0111] Figure 3 The visualization results of high-dimensional features extracted using the deep learning network proposed in this invention and then subjected to t-SNE dimensionality reduction are presented in three datasets of mild cognitive impairment. It can be observed that the high-dimensional features extracted by the network of this invention show significant differences between different types of samples, thus effectively achieving the purpose of binary classification.

[0112] Figure 4 This paper demonstrates how the deep learning network proposed in this invention ultimately selects the top 10 most important brain regions as the basis for discrimination in different neuroimaging modalities for each type of mild cognitive impairment. It can be seen that many cognitively related brain regions were selected.

[0113] Table 1 shows a comparison between the method proposed in this invention and the state-of-the-art (SOTA) methods of the past three years. It can be seen that the present invention ultimately achieved an average classification accuracy of 93.3% in the identification of mild cognitive impairment, and the average area under the ROC curve reached 95%. Figure 5 ROC curves are presented for different enhancement ratios.

[0114] Table 1 Comparison of the effects of the method of this invention with other SOTA methods.

[0115]

[0116] References

[0117] [1]Z.Wang,J.Xin,Z.Wang,Y.Yao,Y.Zhao,and W.Qian,“Brain functionalnetwork modeling and analysis based on fMRI:a systematic review,”CognitiveNeurodynamics,vol.15,pp.389–403,Jun.2021.

[0118] [2]E.M.Palacios et al.,“The evolution of white matter microstructuralchanges after mild traumatic brain injury:A longitudinal DTI and NODDIstudy,”ScienceAdvance,vol.6,no.32,Aug.2020.

[0119] [3]Z.-A.Huang,Z.Zhu,C.H.Yau,and K.C.Tan,“Identifying autismspectrumdisorder from resting-state fMRI using deep belief network,”IEEETransactions onNeural Networks and Learning Systems,vol.32,no.7,pp.2847–2861,Jul.2021.

[0120] [4]B.Iravani,A.Arshamian,P.Fransson,and N.Kaboodvand,“Wholebrainmodelling of resting state fMRI differentiates ADHD subtypes andfacilitates stratifiedneuro-stimulation therapy,”NeuroImage,vol.231,May 2021,Art.no.117844.

[0121] [5]B.Lei et al.,“Multi-scale enhanced graph convolutional network formildcognitive impairment detection,”Pattern Recognition,vol.134,Feb.2023,Art.no.109106.

[0122] [6]N.S.D’Souza et al.,“Deep sr-DDL:Deep structurally regularizeddynamicdictionary learning to integrate multimodal and dynamic functionalconnectomics datafor multidimensional clinical characterizations,”NeuroImage,vol.241,Nov.2021,Art.no.118388.

[0123] [7]J.Sui,R.Huster,Q.Yu,J.M.Segall,and V.D.Calhoun,“Function–structureassociations of the brain:evidence from multimodal connectivity andcovariancestudies,”NeuroImage,vol.102,pp.11-23,Nov.2014.

[0124] [8]S.Sarubbo et al.,“Mapping critical cortical hubs and white matterpathways bydirect electrical stimulation:an original functional atlas of thehuman brain,”NeuroImage,vol.205,Jan.2020,Art.no.116237.

[0125] [9]X.Song et al.,“Multicenter and Multichannel Pooling GCN for EarlyADDiagnosis Based on Dual-Modality Fused Brain Network,”IEEE TransactionsonMedical Imaging,vol.42,no.2,pp.354-367,Feb.2023.

[0126]

[10] J.Huang,L.Zhou,L.Wang and D.Zhang,“Attention-Diffusion-BilinearNeural Network for Brain Network Analysis,”IEEE Transactions onMedical Imaging,vol.39,no.7,pp.2541-2552,July 2020.

[0127]

[11] Q.Zuo,N.Zhong,Y.Pan,H.Wu,B.Lei and S.Wang,“BrainStructure-Function Fusing Representation Learning Using AdversarialDecomposed-VAE forAnalyzing MCI,”IEEE Transactions on Neural Systems andRehabilitationEngineering,vol.31,pp.4017-4028,2023.

[0128]

[12] X.Song et al.,“Graph convolution network with similarityawareness and adaptive calibration for disease-induced deteriorationprediction,”Medical Image Analysis,vol.69,Apr.2021,Art.no.101947.

[0129]

[13] X.Wang,D.Bo,C.Shi,S.Fan,Y.Ye and P.S.Yu,“A Survey onHeterogeneous Graph Embedding:Methods,Techniques,Applications and Sources,”IEEE Transactions on Big Data,vol.9,no.2,pp.415-436,1April 2023.

[0130]

[14] H.Ji,X.Wang,C.Shi,B.Wang and P.S.Yu,“Heterogeneous GraphPropagation Network,”IEEE Transactions on Knowledge and Data Engineering,vol.35,no.1,pp.521-532,1Jan.2023.

[0131]

[15] C.-G.Yan,X.-D.Wang,X.-N.Zuo,and Y.-F.Zang,“DPABI:Data processing&analysis for(resting-state)brain imaging,”Neuroinformatics,vol.14,no.3,pp.339–351,Jul.2016.

[0132]

[16] Z.Cui,S.Zhong,P.Xu,Y.He and G.Gong,“PANDA:A pipeline toolbox foranalyzing brain diffusion images,”Frontiers in Human Neuroscience,vol.7,pp.16,Feb.2013.

[0133]

[17] C.Wu,F.Wu,Y.Huang,X.Xie,“User-as-Graph:User Modeling withHeterogeneous Graph Pooling for News Recommendation,”in IJCAI,2021,pp.1624-1630.

[0134]

[18] W.Li, L.Ni, JW, CW, "Collaborative representation learning fornodes and relations via heterogeneous graph neural network," Knowledge-BasedSystems, vol.255, Nov.2022, Art.no.109673.

[0135]

[19] M.Li, Y.Zhang, W.Zhang, Y.Chu, Y.Hu and B.Yin, "Self-Supervised Nodes-Hyperedges Embedding for Heterogeneous Information Network Learning," IEEETransactions on Big Data, vol.9, no.4, pp.1210-1224, Aug.2023.

[0136]

[20] J.Lee, I.Lee, J.Kang, "Self-Attention Graph Pooling," in ICML, 2019, pp.3734-3743.

[0137]

[21] Z.Ying, J.You, C.Morris,

[0138] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks, characterized in that, The method includes the following steps: S1. Acquire resting-state functional magnetic resonance imaging and diffusion tensor imaging, and construct heterogeneous maps based on the images. The heterogeneous graph includes two different types of nodes: functional brain region type nodes. and structural brain region type nodes The heterogeneous graph includes four types of meta-paths, namely the first meta-path. Secondary path Third Path Fourth Path Wherein, Φ1 and Φ2 are isomorphic paths, while Φ3 and Φ4 are heteromorphic paths; calculate the node features corresponding to functional brain region nodes and structural brain region nodes respectively; S2. Based on the structure-function coupling common community search, the adjacency matrix of four meta-paths is constructed. Based on the adjacency matrix of the four meta-paths, the adjacency matrix of the heterogeneous graph is obtained. The node feature matrix of the heterogeneous graph is obtained by concatenating the node feature matrix of the functional brain region type and the node feature matrix of the structural brain region type. S3, For the first-order path The metapath is reconstructed using a dynamic sliding window method. Simultaneously, the reconstructed metapath is obtained. and Update metapath and The adjacency matrix forms a new heterogeneous graph. The adjacency matrix, the node feature matrix of the new heterogeneous graph, and the heterogeneous graph before reconstruction. The node feature matrix remains consistent; S4, Heterogeneous Diagram and new heterogeneous graphs The adjacency matrix and node feature matrix are input together into the mild cognitive impairment identification model. The mild cognitive impairment identification model is constructed based on the heterogeneous graph attention neural network (HAN), which includes an improved heterogeneous graph attention pooling layer. The mild cognitive impairment identification model outputs the identification result.

2. The automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks according to claim 1, characterized in that, The adjacency matrix of the first metapath is: in, This represents the i-th row and j-th column of the adjacency matrix of the first-order path. This represents the node feature of node i in the functional brain region type node, where d represents the dimension of the corresponding node feature, and E i σ represents the mean of the features of node i. i Cov represents the variance of the features of node i. ij This represents the covariance between node i and node j in a functional brain region type node.

3. The automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks according to claim 2, characterized in that, The calculation process of the adjacency matrix of the second-ary path is as follows: deterministic white matter fiber tract tracking is performed between every two regions of interest (ROIs). The number of tracked white matter fiber tracts is used as the edge weight between the nodes of these two structural brain regions, which is the i-th row and j-th column of the adjacency matrix of the second-ary path. When the edge weight is less than the threshold, it is discarded. The structural brain region type node is a region of interest (ROI) divided by diffusion tensor imaging.

4. The automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks according to claim 3, characterized in that, The calculation process for the adjacency matrix of the third-order path and the adjacency matrix of the fourth-order path is as follows: For a pair of cross-modal nodes Their connection modes in their respective modes are as follows: and Calculate the cosine similarity ε between these two connection modes. ij For each node i of a functional brain region type node and a node of a structural brain region type node, the cosine similarity between their connection patterns is calculated to obtain a vector. In vector The values ​​of the 8 most similar connection patterns are selected and used as the i-th row of the first adjacency matrix. The value at the corresponding position in the middle; A common community search based on structure-function coupling is adopted to find a closed subgraph g consisting of 3 nodes. The closed subgraph g satisfies the condition that it exists in both Φ1 and Φ2. The second adjacency matrix is ​​calculated based on the closed subgraph g. Add the first adjacency matrix and the second adjacency matrix to obtain the adjacency matrix of the third path; Transpose the adjacency matrix of the third-order path to obtain the adjacency matrix of the fourth-order path.

5. The automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks according to claim 4, characterized in that, The first adjacency matrix is: in, Represents the i-th row and j-th column of the first adjacency matrix, TopK j This indicates selecting the indices of the K largest elements in the vector; here, K is set to 8.

6. The automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks according to claim 5, characterized in that, The second adjacency matrix is: in, It represents the i-th row and j-th column of the second adjacency matrix.

7. The automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks according to claim 1, characterized in that, The specific steps for S3 are as follows: A sliding window method is used to perform a sliding window operation on the node features corresponding to nodes in functional brain regions. Within each window, multiple local meta-paths are constructed using the Pearson correlation coefficient, considering the inclusion of edges. Three-node subgraph g ij ,side This represents an edge between two functional brain region type nodes. Weights are assigned to this edge in three different types of subgraphs corresponding to the three possible edge scenarios in the subgraph. Count the three types of subgraphs g containing this edge. ij The total number of occurrences in a local metapath is denoted by a vector. The three edge connection cases are: containing only edges Includes edges And any other edge, and containing The closed subgraph formed by the other two edges; The metapath obtained by computational reconstruction adjacency matrix Keeping the adjacency matrix of the second path Φ2 unchanged, based on the adjacency matrix Update metapath using the S2 method and The adjacency matrix forms a new heterogeneous graph. and new heterogeneous graphs First path Secondary path Φ2, tertiary path Fourth Path and 8. The automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks according to claim 7, characterized in that, Adjacency Matrix for: in, Indicates the presence of edges Three types of three-node subgraphs g ij The total number of occurrences in the local metapath This represents the transpose of the weights.

9. The automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks according to claim 1, characterized in that, The heterogeneous graph attention neural network (HAN) includes three heterogeneous graph attention convolutional layers, three improved heterogeneous graph attention pooling layers, and three graph readout layers.

10. The automatic identification method for mild cognitive impairment based on heterogeneous graph neural networks according to claim 9, characterized in that, The computational flow within the improved heterogeneous graph attention pooling layer is as follows: For the heterogeneous graph output of the l-th layer heterogeneous graph attention convolutional layer of the network, the attention score matrix of each node type is calculated, and then the pooling matrix is ​​calculated. The pooling matrix is ​​applied to the node feature matrix and adjacency matrix of the heterogeneous graph. The result of the pooling is used as the output of the improved heterogeneous graph attention pooling layer. The attention score matrix is ​​as follows: Among them, S (l) Represents the attention score matrix, θ (l) These represent the input adjacency matrix, input node feature matrix, and parameter matrix of the current layer received by the l-th layer of the neural network, respectively. The subscript f corresponds to the functional state, and the subscript d corresponds to the structural state. The pooling matrix is: Among them, P (l) D represents the final pooling matrix after unifying the dimensions using an all-zero matrix. (l) Let O represent the dense pooling information matrix to be unified in terms of dimension, and let b represent the matrix of all zeros. (l) W represents the bias vector in the linear layer. (l) The parameter matrix represents the linear layer, with the subscript f corresponding to the functional state and the subscript d corresponding to the structural state. The result after pooling is: in, This represents the result after pooling. These represent the adjacency matrices corresponding to the first, third, and second ary paths in the heterogeneous graph of the input neural network, respectively.

Images