A classification method for multimodal brain hypergraph attention networks based on line graph extensions

Abstract

This invention discloses a classification method for a multimodal brain hypergraph attention network based on line graph extension. After preprocessing structural and functional magnetic resonance imaging (MRI) images, structural and functional features are extracted from the MRI images of each subject. After feature fusion, a multimodal hypergraph is generated using sparse linear regression. The multimodal hypergraph correlation matrix and multimodal feature matrix are input into the hypergraph attention network. Utilizing the transformation between line graphs and hypergraphs, nodes and hyperedges of the hypergraph alternate in each layer of the neural network, achieving simultaneous integration of features from both hypernodes and hyperedges for node and edge embedding learning. A two-layer attention mechanism is used to learn different weights for hyperedges and nodes. The extracted features are finally processed through a softmax operation and classified using a cross-entropy loss function. This invention is applied to brain disease diagnosis, preserving the connectivity of hyperedges in the multimodal hypergraph while utilizing edge-to-edge relationships to propagate information, thus facilitating the overall feature learning of the hypergraph.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to a classification method for a multimodal brain hypergraph attention network based on line graph extension. Background Technology

[0002] The human brain is a highly complex system, and exploring its internal structure and function is a tremendous challenge. The combination of magnetic resonance imaging (MRI) and hypergraph attention networks has become a research hotspot in neuroscience and is widely used in various studies. This method has achieved many surprising results in exploring the similarity of whole-brain structures in subjects. However, due to limitations in its own principles, characteristics, and methodology, the classification accuracy is low, severely impacting its application value.

[0003] In traditional resting-state functional magnetic resonance imaging (fMRI) classification studies, combining MRI image information with machine learning algorithms can create classifiers and predictors for various diseases. While machine learning, as an important analytical tool, has provided new perspectives in the field of brain disease diagnosis, some challenges remain.

[0004] First, machine learning fails to consider the relationships between subjects. Second, in traditional methods, feature representations based on regions of interest are insufficient to represent all the information of the subjects, and feature selection is independent of subsequent classifiers, which may not coordinate well with subsequent classifiers, potentially leading to a decline in diagnostic performance.

[0005] Based on this, researchers proposed a deep learning framework, in which graph neural networks (GNNs) are particularly suitable for neuroimaging research due to their ability to represent unstructured information. In recent years, GNNs have been widely used in non-invasive brain imaging techniques for the diagnosis and prediction of brain diseases, and have seen rapid development. However, the input to graph convolutional neural network methods is a simple graph, where subjects are connected in pairs. In reality, however, the unpaired relationships between subjects cannot be represented by simple graphs. Traditional simple graph structures have limitations in representing correlations, restricting the application of GNNs. Therefore, hypergraph neural networks were proposed and applied to the classification and diagnosis of brain diseases.

[0006] However, existing hypergraph neural networks can only obtain effective node embeddings by aggregating information from neighboring nodes, while ignoring edge-related information. Furthermore, in the embedding learning process, existing research assumes that the hyperedges of the hypergraph are decomposable, explicitly or implicitly transforming the hyperedges into simpler graphs through star-shaped expansion or connected component expansion, and then performing downstream mining tasks on the transformed simpler graphs. However, the relations that exist within a complete hyperedge may not exist in an incomplete subset of that hyperedge, which leads to the loss of information from high-dimensional data and is not conducive to overall feature learning. Summary of the Invention

[0007] This invention provides a classification method for a multimodal brain hypergraph attention network based on line graph extension. It utilizes sparse representation learning to measure the similarity between subjects to construct a hypergraph. The multimodal brain hypergraph attention network model based on line graph extension can use line graphs and alternate between supernodes and superedges in each layer of the neural network. Based on the alternating stacking of two-layer attention mechanisms, it can simultaneously combine the features of supernodes and superedges to perform node and edge embedding learning.

[0008] The technical solution adopted in this invention is a classification method based on a multimodal brain hypergraph attention network with line graph extension, which is carried out according to the following steps:

[0009] Step S1: Preprocess the structural magnetic resonance imaging (SMRI) and functional magnetic resonance imaging (fMRI) images of the subject to obtain preprocessed SMRI and fMRI images;

[0010] Step S2: Based on the selected standardized brain atlas, perform brain region segmentation on the preprocessed structural magnetic resonance imaging (SMRI) and functional magnetic resonance imaging (fMRI) to obtain segmented SMRI and fMRI images.

[0011] Step S3: Extract structural features from the segmented structural magnetic resonance images to obtain a structural feature matrix; extract the average time series from the segmented functional magnetic resonance images, construct a functional brain network, and then extract functional features to obtain a functional feature matrix; fuse the functional feature matrix and the structural feature matrix to obtain fused multimodal features.

[0012] Step S4: Based on the fused multimodal features, construct a multimodal hypergraph using the sparse linear regression method to obtain a dataset, and divide the dataset into a training set and a test set. The dataset includes a multimodal hypergraph association matrix, a hypergraph node feature matrix, and node labels.

[0013] Step S5: Based on the multimodal hypergraph association matrix, use row normalization and column normalization to form the multimodal line graph adjacency matrix;

[0014] Step S6: Based on the multimodal line graph adjacency matrix, hierarchical two-layer attention mechanism, softmax function, and cross-entropy loss function, construct a multimodal hypergraph attention network model using the line graph expansion method, and train and optimize it using the training set;

[0015] Step S7: Use cross-validation to test the constructed multimodal hypergraph attention network model.

[0016] In step S1, the structural and functional magnetic resonance imaging data of the subject are preprocessed, including: time correction, head motion correction, registration and spatial normalization, and low-frequency filtering.

[0017] In step S2, brain region segmentation is performed on the images based on the selected standardized brain atlas. Specifically, this involves selecting the internationally recognized Automated Anatomical Labeling (AAL) template as the standardized brain atlas and segmenting the preprocessed image data into brain regions. The template divides the whole brain into 90 brain regions, with 45 regions in each hemisphere.

[0018] In step S3, structural features are extracted from the segmented structural magnetic resonance images. Specific steps include cranial separation and gray-white matter separation. First, a nonparametric non-uniform intensity normalization algorithm is used to correct the intensity non-uniformity. Then, the watershed algorithm is used to separate the cranial structure of the obtained image into subcortical white matter and gray matter volume structures. After separating the gray and white matter of the image, the cortical thickness and gray matter volume of each brain region are calculated based on the generated high-precision gray and white matter segmentation surfaces. All brain region features extracted from each image constitute a feature vector, thereby obtaining the brain region structural feature matrix of all subjects.

[0019] Then, the average time series of the segmented functional magnetic resonance imaging is extracted. The specific steps include: extracting the BOLD intensity of all voxels contained in each brain region at different time points, and then performing an arithmetic average to obtain the average time series of the brain region. Based on the average time series of the brain region, the linear combination representation of each brain region with other brain regions is obtained. The functional brain network is constructed and the properties of the functional brain network are calculated, including node degree, hyperedge degree, and node efficiency, and then the functional feature matrix of each subject is obtained.

[0020] Finally, feature fusion is performed on the functional and structural feature matrices. The specific steps are as follows: the obtained structural features and functional features are connected.

[0021] In step S4, based on the fused multimodal features, a multimodal hypergraph dataset is constructed using the sparse linear regression method. Specifically, each participant node is used as a centroid node, and hyperedges are constructed by connecting the centroid nodes to other nodes. The sparsity coefficient is used as a similarity measure. The sparse linear regression model is specifically represented as follows:

[0022]

[0023] In formula (1): s m The weight vector x represents the correlation between the centroid node and other subject nodes. m F represents the multimodal features of the selected m-th subject node. m = [x1, ..., x m-1 ,0,x m+1 , ..., x M[Includes multimodal features of all participants except the selected participants, ||·||2 represents the l2 norm, λ represents the regularization parameter controlling model sparsity, and ||·||1 represents the l1 norm, based on x] m F m We can get s m s m The subjects corresponding to the non-zero elements are those that are relevant to the selected subjects, thus obtaining the hyperedge.

[0024] A multimodal hypergraph is constructed to obtain a dataset, which includes a multimodal hypergraph association matrix, a hypergraph node feature matrix, and node labels. The specific steps for constructing the multimodal hypergraph are as follows: using the weight vector s obtained in formula (1) to determine the correlation between the m-th centroid node and other subject nodes. m The weight matrix S of a sparse hypergraph is defined as S = [s1, s2, ..., s]. m , ..., s M ] T A threshold θ is selected, which is set to 0.001 in this study. Nodes with similarity greater than this threshold indicate connectivity, while those less than the threshold indicate no connectivity. Nodes with high similarity are connected by a hyperedge, and the hypergraph association matrix is ​​represented as follows:

[0025]

[0026] In formula (2), H(v) i e j ) represents the corresponding element in the multimodal hypergraph association matrix, v∈V represents a node, e∈E represents a hyperedge, θ represents the threshold, and S i，j Represents node v i In the super-edge e j Weighting on.

[0027] The hypergraph node feature matrix in the dataset represents the set of structural features and brain functional network features of each brain region in the subject's brain, denoted as X; the dataset is divided into training set and test set according to the subject's information (label, site, gender).

[0028] In step S5, a line graph adjacency matrix is ​​formed based on the multimodal hypergraph association matrix using row and column normalization. The introduction of the line graph is based on the fact that most existing hypergraph neural networks only achieve feature updates by aggregating node information, ignoring the structural information of hyperedges in the hypergraph. Introducing the line graph allows the hypergraph to retain its hyperedge structural information and achieve hyperedge information aggregation during learning in the neural network model. The specific steps are: converting the hyperedges of the hypergraph into nodes of the line graph; if two hyperedges share a node, then connecting the two nodes in the corresponding line graph. The line graph adjacency matrix corresponding to the hypergraph contains information about how the nodes of the line graph are connected. Simultaneously, to avoid excessively high weights in the line graph for connecting two higher-order hyperedges, row and column normalization of the association matrix is ​​used to form the multimodal line graph adjacency matrix. The specific formula is as follows:

[0029]

[0030] In formula (3), A L D represents the weighted form of the adjacency matrix of a multimodal line graph. E H represents the hyperedge degree matrix of a multimodal hypergraph. T D represents the transpose of the multimodal hypergraph incidence matrix. V Let H represent the degree matrix of the hypernodes in the multimodal hypergraph, and let H represent the association matrix of the multimodal hypergraph.

[0031] In step S6, a multi-layer attention mechanism based on hierarchical structure is used. The softmax function and cross-entropy loss function are employed to construct a multimodal hypergraph attention network model using the line graph expansion method, and the model is then trained and optimized on the training set. Specifically, in most cases, hypergraph neural networks assume that all hyperedges in the hypergraph are of equal size. However, for the final classification task, the existence of some hyperedges may be more important than others. Introducing a two-layer attention mechanism allows for learning the importance of the neighbors of nodes and hyperedges at both the node and hyperedge levels. This enables the generation of effective node and hyperedge embeddings based on node and hyperedge features, thereby significantly improving the expressive power of the hypergraph neural network model.

[0032] First, the training set is input into the hypergraph attention network model, which consists of two stacked attention networks. Each layer is further divided into two layers based on the node layer and the hyperedge layer. The hierarchical multi-layer attention mechanism can learn different weights for different nodes and hyperedges from the attention mechanisms of the node layer and the hyperedge layer, respectively. The specific representation of the attention mechanism of the node layer is as follows:

[0033]

[0034] In formula (4), W V σ represents the weight of the node obtained based on the updated hyperedge features at each layer. attLet HX represent a non-linear activation function. l-1 This represents the hypergraph node feature matrix obtained from the update of layer l-2, where, Let x represent the feature matrix of the M subjects obtained from the (l-1)th layer update. m H represents the brain region features of the selected m-th subject, and H represents the hypergraph association matrix. It is the attention vector.

[0035] The specific representation of the attention mechanism of the super-edge layer is as follows:

[0036]

[0037] In formula (5), W E σ represents the weight of the hyperedge obtained based on the node features updated at each layer. att H represents a non-linear activation function. T X l-1 This represents the feature matrix of the line graph nodes obtained from the (l-2)th layer update, where, Let x represent the feature matrix of the M subjects obtained from the (l-1)th layer update. m H represents the brain region features of the selected m-th subject, and H represents the hypergraph association matrix. It is the attention vector.

[0038] Then, a multimodal hypergraph attention network model is constructed using the line graph expansion method. The specific steps are as follows: the node feature matrix of the line graph in the (l-1)th layer comes from the updated hypergraph node features in this layer. In the (l-1)th layer, the hyperedge features aggregated by the line graph are redistributed to the nodes of the hypergraph and passed to the next layer.

[0039] The feature information of the hypergraph nodes in layer (l-1) is mapped to the node feature matrix of the line graph in layer l in the form of matrix multiplication. Based on the attention mechanism of the hyperedge layer, the hyperedge weights are assigned to the line graph nodes. The line graph runs a hyperedge aggregation network to obtain hyperedge information and obtain the updated feature matrix. The specific formula is as follows:

[0040]

[0041] In formula (6), Let A represent the feature matrix of the (l-1)th layer obtained by aggregating hyperedge information. L This represents the weighted adjacency matrix of the line graph, where D(·) denotes the diagonal operation, W represents the weight of the hyperedge in the hypergraph obtained based on the node features updated at each layer, and H represents the weight of the hyperedge. T X l-1 This represents the node feature matrix updated from layer l-2, where, Let x represent the feature matrix of the M subjects obtained from the (l-1)th layer update. m H represents the brain region features of the selected m-th subject, and H represents the hypergraph association matrix. It is a trainable parameter.

[0042] The updated node features of the midline graph in layer (l-1) are mapped to the hypergraph in layer l. Node weights are learned based on the attention mechanism of the node layers. The node feature aggregation network obtains node embeddings by aggregating information from neighboring nodes according to the weights, and updates the feature matrix. The specific representation is as follows:

[0043]

[0044] In formula (7), X l D represents the l-th layer feature matrix obtained by updating the aggregated node information, D(·) represents the diagonal operation, and W V H represents the weights of nodes in the hypergraph obtained based on the updated hyperedge features at each layer, and H represents the hypergraph association matrix. This represents the feature matrix of the (l-1)th layer obtained by aggregating hyperedge information. Q is a trainable parameter.

[0045] Finally, the obtained supernode features are used to classify nodes through a softmax layer. The cross-entropy loss function is then used on the training set to calculate the loss value, and stochastic gradient descent is used to optimize the cross-entropy error, thereby improving the model's classification ability. The specific expression of the softmax function is as follows:

[0046]

[0047] In formula (8), v represents the probability that the label of the i-th node is k. i This represents the i-th node, where e is the Napier constant;

[0048] The cross-entropy loss function is expressed as follows:

[0049]

[0050] In formula (9), Represents hypergraph node v i The actual label is k. Represents hypergraph node v i The probability that the label is k in the output label of the softmax function;

[0051] In step S7, the cross-validation method is used to test the constructed multimodal hypergraph attention network. The specific steps are as follows: 90% of the samples in the dataset are randomly selected as the training set, and the remaining 10% are selected as the test set. Classification tests are performed and the classification accuracy is obtained. The classification accuracy obtained after repeating cross-validation 100 times is arithmetically averaged, and then the arithmetic mean is used as the classification accuracy of the classifier.

[0052] The beneficial effects of this invention are as follows: a multimodal hypergraph is constructed using a sparse representation method, and then the transformation relationship between the line graph and the hypergraph is utilized to make the hypernodes and hyperedges alternate in each layer of the neural network. Based on the alternating stacking of the two-layer attention mechanism, the embedding learning of nodes and edges is achieved by combining the features of hypernodes and hyperedges at the same time. This not only preserves the integrity of the hypergraph structure, but also utilizes the edge-to-edge relationship to propagate information, presenting a higher level and more complex relationship between subjects. Attached Figure Description

[0053] Figure 1 This is a flowchart of the multimodal hypergraph attention network classification method based on line graph extension implemented in this invention.

[0054] Figure 2 This is a diagram of a multimodal hypergraph attention network model based on line graph extension, as implemented in this invention.

[0055] Figure 3 This is a schematic diagram comparing the present invention with the graph neural network magnetic resonance imaging data classification method. Detailed Implementation

[0056] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0057] like Figure 1-2 As shown, the classification method based on the multimodal brain hypergraph attention network extended from the line graph is carried out according to the following steps:

[0058] Step S1: Preprocess the structural magnetic resonance imaging (SMRI) and functional magnetic resonance imaging (fMRI) images of the subject to obtain preprocessed SMRI and fMRI images;

[0059] Step S2: Based on the selected standardized brain atlas, perform brain region segmentation on the preprocessed structural magnetic resonance imaging (SMRI) and functional magnetic resonance imaging (fMRI) to obtain segmented SMRI and fMRI images.

[0060] Step S3: Extract structural features from the segmented structural magnetic resonance images to obtain a structural feature matrix; extract the average time series from the segmented functional magnetic resonance images, construct a functional brain network, and then extract functional features to obtain a functional feature matrix; fuse the functional feature matrix and the structural feature matrix to obtain fused multimodal features.

[0061] Step S4: Based on the fused multimodal features, construct a multimodal hypergraph using the sparse linear regression method to obtain a dataset, and divide the dataset into a training set and a test set. The dataset includes a multimodal hypergraph association matrix, a hypergraph node feature matrix, and node labels.

[0062] Step S5: Based on the multimodal hypergraph association matrix, use row normalization and column normalization to form the multimodal line graph adjacency matrix;

[0063] Step S6: Based on the multimodal line graph adjacency matrix, hierarchical two-layer attention mechanism, softmax function, and cross-entropy loss function, construct a multimodal hypergraph attention network model using the line graph expansion method, and train and optimize it using the training set;

[0064] Step S7: Use cross-validation to test the constructed multimodal hypergraph attention network model.

[0065] In step S1, the structural and functional magnetic resonance imaging data of the subject are preprocessed, including time correction, head motion correction, registration and spatial normalization, and low-frequency filtering.

[0066] In step S2, brain region segmentation is performed on the images based on the selected standardized brain atlas. Specifically, this involves selecting the internationally recognized Automated Anatomical Labeling (AAL) template as the standardized brain atlas and segmenting the preprocessed image data into brain regions. The template divides the whole brain into 90 brain regions, with 45 regions in each hemisphere.

[0067] In step S3, structural features are extracted from the segmented structural magnetic resonance images. Specific steps include cranial separation and gray-white matter separation. First, a nonparametric non-uniform intensity normalization algorithm is used to correct the intensity non-uniformity. Then, the watershed algorithm is used to separate the cranial structure of the obtained image into subcortical white matter and gray matter volume structures. After separating the gray and white matter of the image, the cortical thickness and gray matter volume of each brain region are calculated based on the generated high-precision gray and white matter segmentation surfaces. All brain region features extracted from each image constitute a feature vector, thereby obtaining the brain region structural feature matrix of all subjects.

[0068] The average time series of the segmented functional magnetic resonance imaging (fMRI) images is extracted. The specific steps include: extracting the BOLD intensity of all voxels contained in each brain region at different time points, and then performing an arithmetic average to obtain the average time series of the brain region. Based on the average time series of the brain region, a linear combination representation of each brain region with other brain regions is obtained. A functional brain network is constructed and the properties of the functional brain network are calculated, including node degree, hyperedge degree, and node efficiency, thereby obtaining the functional feature matrix of each subject.

[0069] The functional and structural feature matrices are fused using the following steps: the obtained structural and functional features are connected.

[0070] In step S4, based on the fused multimodal features, a multimodal hypergraph dataset is constructed using the sparse linear regression method. Specifically, each participant node is used as a centroid node, and hyperedges are constructed by connecting the centroid nodes to other nodes. The sparsity coefficient is used as a similarity measure. The sparse linear regression model is specifically represented as follows:

[0071]

[0072] In formula (1): s m The weight vector, x, represents the correlation between other subjects and the selected subject. m F represents the multimodal feature of the selected m-th subject. m = [x1, ..., x m-1 ,0,x m+1 , ..., x M [Includes multimodal features of all participants except the selected participants, ||·||2 represents the l1 norm, λ represents the regularization parameter controlling model sparsity, and ||·||1 represents the l2 norm, based on x] m F m We can get s m s m The subjects corresponding to the non-zero elements are those that are relevant to the selected subjects, thus obtaining the hyperedge.

[0073] A multimodal hypergraph is constructed to obtain a dataset, which includes a multimodal hypergraph association matrix, a hypergraph node feature matrix, and node labels. The specific steps for constructing the multimodal hypergraph are as follows: using the weight vector s between subjects obtained in formula (1) m The sparse representation weight matrix S is defined as S = [s1, s2, ..., s]. m , ..., s M ] TA threshold θ is selected, which is set to 0.001 in this study. Nodes with similarity greater than this threshold indicate connectivity, while those less than the threshold indicate no connectivity. Connecting nodes with high similarity using a hyperedge, the multimodal hypergraph association matrix is ​​defined as follows:

[0074]

[0075] In formula (2), H(v) i e j ) represents the corresponding element in the hypergraph incidence matrix, v∈y represents a node, e∈E represents a hyperedge, θ represents the threshold, and S i，j Represents node v i In the super-edge e j Weighting on.

[0076] The hypergraph node feature matrix in the dataset represents the set of structural features and brain functional network features of each brain region in the subject's brain, denoted as X; the dataset is divided into training set and test set according to the subject's information (label, site, gender).

[0077] In step S5, a multimodal line graph adjacency matrix is ​​formed based on the multimodal hypergraph association matrix using row and column normalization. The introduction of the line graph is based on the fact that most existing hypergraph neural networks only achieve feature updates by aggregating node information, ignoring the structural information of hyperedges in the hypergraph. Introducing the line graph allows the hypergraph to retain its hyperedge structural information and achieve hyperedge information aggregation during learning in the neural network model. Specifically, the hyperedges of the hypergraph are transformed into nodes of the line graph. If two hyperedges share a node, the two nodes in the corresponding line graph are connected. Thus, the adjacency matrix of the line graph corresponding to the hypergraph contains information about how the nodes of the line graph are connected. To avoid the weights connecting two higher-order hyperedges being too high in the line graph, row and column normalization of the association matrix are used to form the adjacency matrix of the multimodal line graph, as shown in the following formula:

[0078]

[0079] In formula (3), A L D represents the weighted form of the adjacency matrix of a multimodal line graph. E H represents the hyperedge degree matrix of a multimodal hypergraph. T D represents the transpose of the multimodal hypergraph incidence matrix. V Let H represent the degree matrix of the hypernodes in the multimodal hypergraph, and let H represent the association matrix of the multimodal hypergraph.

[0080] In step S6, a multi-layer attention mechanism based on hierarchical structure is used. The softmax function and cross-entropy loss function are employed to construct a multimodal hypergraph attention network model using the line graph expansion method, and the model is then trained and optimized on the training set. Specifically, in most cases, hypergraph neural networks assume that all hyperedges in the hypergraph are of equal size. However, for the final classification task, the existence of some hyperedges may be more important than others. Introducing a two-layer attention mechanism allows for learning the importance of the neighbors of nodes and hyperedges at both the node and hyperedge levels. This enables the generation of effective node and hyperedge embeddings based on node and hyperedge features, thereby significantly improving the expressive power of the hypergraph neural network model.

[0081] First, the training set is input into the hypergraph attention network model, which consists of two stacked attention networks. Each layer is further divided into two layers based on the node layer and the hyperedge layer. The hierarchical multi-layer attention mechanism can learn different weights for different nodes and hyperedges from the attention mechanisms of the node layer and the hyperedge layer, respectively. The specific representation of the attention mechanism of the node layer is as follows:

[0082]

[0083] In formula (4), W V σ represents the weight of the node obtained based on the updated hyperedge features at each layer. att Let HX represent a non-linear activation function. l-1 This represents the hypergraph node feature matrix updated from layer l-2, where, Let x represent the feature matrix of the M subjects obtained from the (l-1)th layer update. m Let H represent the multimodal features of the selected m-th subject, and let H represent the multimodal hypergraph correlation matrix. It is the attention vector.

[0084] The specific representation of the attention mechanism of the super-edge layer is as follows:

[0085]

[0086] In formula (5), W E σ represents the weight of the hyperedge obtained based on the node features updated at each layer. att H represents a non-linear activation function. T X l-1 This represents the feature matrix of the line graph nodes obtained from the (l-2)th layer update, where, Let x represent the feature matrix of the M subjects obtained from the (l-1)th layer update. m H represents the brain region features of the selected m-th subject, and H represents the hypergraph association matrix. It is the attention vector.

[0087] Then, a multimodal hypergraph attention network model is constructed using the line graph expansion method. The specific steps are as follows: the node feature matrix of the line graph in the (l-1)th layer comes from the updated hypergraph node features in this layer. In the (l-1)th layer, the hyperedge features aggregated by the line graph are redistributed to the nodes of the hypergraph and passed to the next layer.

[0088] The feature information of the hypergraph nodes in layer (l-1) is mapped to the node feature matrix of the line graph in layer l in the form of matrix multiplication. Based on the attention mechanism of the hyperedge layer, the hyperedge weights are assigned to the line graph nodes. The line graph runs a hyperedge aggregation network to obtain hyperedge information and obtain the updated feature matrix. The specific formula is as follows:

[0089]

[0090] In formula (6), Let A represent the feature matrix of the (l-1)th layer obtained by aggregating hyperedge information. L This represents the weighted form of the adjacency matrix of a line graph, where D(·) denotes diagonal operation, and W... E H represents the weight of the hyperedge in the hypergraph obtained based on the node features updated at each layer. T X l-1 This represents the node feature matrix updated from layer l-2, where, Let x represent the feature matrix of the M subjects obtained from the (l-1)th layer update. m H represents the brain region features of the selected m-th subject, and H represents the hypergraph association matrix. It is a trainable parameter.

[0091] The updated node features of the midline graph in layer (l-1) are mapped to the hypergraph in layer l. Node weights are learned based on the attention mechanism of the node layers. The node feature aggregation network obtains node embeddings by aggregating information from neighboring nodes according to the weights, and updates the feature matrix. The specific representation is as follows:

[0092]

[0093] In formula (7), X l D represents the l-th layer feature matrix obtained by updating the aggregated node information, D(·) represents the diagonal operation, and W V H represents the weights of nodes in the hypergraph obtained based on the updated hyperedge features at each layer, and H represents the hypergraph association matrix. This represents the feature matrix of the (l-1)th layer obtained by aggregating hyperedge information. Q is a trainable parameter. Finally, the obtained supernode features are used to classify nodes through a softmax layer. The cross-entropy loss function is then used on the training set to calculate the loss value, and stochastic gradient descent is used to optimize the cross-entropy error, thereby improving the model's classification ability. The specific expression of the softmax function is as follows:

[0094]

[0095] In formula (8), v represents the probability that the label of the i-th node is k. i This represents the i-th node, where e is the Napier constant;

[0096] The cross-entropy loss function is expressed as follows:

[0097]

[0098] In formula (9), Represents hypergraph node v i The actual label is k. Represents hypergraph node v i The probability that the label is k in the output label of the softmax function;

[0099] In step S7, the cross-validation method is used to test the constructed multimodal hypergraph attention network. The specific steps are as follows: 90% of the samples in the dataset are randomly selected as the training set, and the remaining 10% are selected as the test set. Classification tests are performed and the classification accuracy is obtained. The classification accuracy obtained after repeating cross-validation 100 times is arithmetically averaged, and then the arithmetic mean is used as the classification accuracy of the classifier.

[0100] By employing sparse linear regression, hyperedge information aggregation network, hypernode feature aggregation network, two-layer attention mechanism, softmax activation function, cross-entropy loss function, and cross-validation method, the application of hypergraph attention network in the field of brain diseases was realized, thereby significantly improving the classification accuracy.

[0101] like Figure 3 As shown, the classification accuracy of this invention is significantly higher than that of graph neural networks on magnetic resonance imaging data, thus making its application value higher. This invention overcomes the shortcomings of existing graph neural network methods for magnetic resonance imaging data classification by applying a multimodal hypergraph attention network to the classification of magnetic resonance imaging data.

[0102] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A classification method based on a multimodal brain hypergraph attention network with line graph extension, characterized in that, Includes the following steps: Step S1: Preprocess the structural magnetic resonance imaging (SMRI) and functional magnetic resonance imaging (fMRI) images of the subject to obtain preprocessed SMRI and fMRI images; Step S2: Based on the selected standardized brain atlas, perform brain region segmentation on the preprocessed structural magnetic resonance imaging (SMRI) and functional magnetic resonance imaging (fMRI) to obtain segmented SMRI and fMRI images. Step S3: Extract structural features from the segmented structural magnetic resonance images to obtain a structural feature matrix; extract the average time series from the segmented functional magnetic resonance images, construct a functional brain network, and then extract functional features to obtain a functional feature matrix; fuse the functional feature matrix and the structural feature matrix to obtain fused multimodal features. Step S4: Based on the fused multimodal features, construct a multimodal hypergraph using the sparse linear regression method to obtain a dataset, and divide the dataset into a training set and a test set. The dataset includes a multimodal hypergraph association matrix, a hypergraph node feature matrix, and node labels. Step S5: Based on the multimodal hypergraph association matrix, use row normalization and column normalization to form the multimodal line graph adjacency matrix; Step S6: Based on the multimodal line graph adjacency matrix, hierarchical two-layer attention mechanism, softmax function, and cross-entropy loss function, construct a multimodal hypergraph attention network model using the line graph expansion method, and train and optimize it using the training set; Step S7: Use cross-validation to test the constructed multimodal hypergraph attention network model; In step S4, the sparse linear regression method is specifically represented as follows: In formula (1), s m The weight vector, x, represents the correlation between other subjects and the selected subject. m F represents the multimodal feature of the selected m-th subject. m =[x1,…,x m-1 ,0,x m+1 ,…,x M [Includes multimodal features of all participants except the selected participants, ||·||2 represents the l2 norm, λ represents the regularization parameter controlling model sparsity, and ||·||1 represents the l1 norm, based on x] m F m We can get s m s m The subjects corresponding to the non-zero elements are those that are related to the selected subjects, thus obtaining the hyperedge; The specific steps for constructing a multimodal hypergraph are as follows: Based on the weight vector s m The sparse representation weight matrix S is defined as S = [s1, s2, ..., s]. m ,…,s M ] T A threshold θ is selected; nodes greater than the threshold indicate connectivity, and nodes less than the threshold indicate disconnectivity. Nodes with high similarity are connected by a hyperedge. The multimodal hypergraph association matrix is ​​defined as follows: In formula (2), H(v) i ,e j ) represents the corresponding element in the multimodal hypergraph incidence matrix, v∈V represents a node, e∈E represents a hyperedge, θ represents the threshold, and S=[s1,s2,…,s m ,…,s M ] T S represents the weight matrix of a multimodal sparse hypergraph. i,j Represents node v i In the super-edge e j Weights on; The hypergraph node feature matrix represents the set of structural features and brain functional network features of each brain region in the subject's brain, denoted as X; the dataset is divided into training set and test set according to the subject's label, site, and gender.

2. The classification method based on a multimodal brain hypergraph attention network with line graph extension as described in claim 1, characterized in that, In step S1, the specific preprocessing steps include: time correction, head motion correction, registration and spatial normalization, and low-frequency filtering.

3. The classification method based on a multimodal brain hypergraph attention network with line graph extension as described in claim 1, characterized in that, In step S3 Structural feature extraction is performed on the segmented structural magnetic resonance images. The specific steps include: First, a nonparametric non-uniform intensity normalization algorithm is used to correct the intensity non-uniformity. Then, the watershed algorithm is used to separate the skull in the obtained image, segmenting it into subcortical white matter and gray matter volume structures. After separating the gray and white matter in the image, the cortical thickness and gray matter volume of each brain region are calculated based on the generated high-precision gray and white matter segmentation surfaces. All brain region features extracted from each image constitute a feature vector, thereby obtaining the brain region structure feature matrix of all subjects. The average time series of the segmented functional magnetic resonance imaging (fMRI) images is extracted, and the specific steps include: The BOLD intensity of all voxels contained in each brain region at different time points is extracted, and then the arithmetic mean is performed to obtain the average time series of the brain region. Based on the average time series of the brain region, the linear combination representation of each brain region with other brain regions is obtained, the functional brain network is constructed, and the properties of the functional brain network are calculated, including: node degree, hyperedge degree, node efficiency, and then the functional feature matrix of each subject is obtained. Feature fusion is performed on the functional and structural feature matrices, and the specific steps include: The obtained structural and functional features are connected.

4. The classification method based on a multimodal brain hypergraph attention network with line graph extension as described in claim 1, characterized in that, Step S5 specifically includes: converting the hyperedges of the hypergraph into nodes of the line graph; if two hyperedges share a node, connecting the two nodes in the corresponding line graph; and using row normalization and column normalization of the incidence matrix to form the adjacency matrix of the multimodal line graph, as shown in the following formula: In formula (3), A L D represents the weighted form of the adjacency matrix of a multimodal line graph. E H represents the hyperedge degree matrix of a multimodal hypergraph. T D represents the transpose of the multimodal hypergraph incidence matrix. V Let H represent the degree matrix of the hypernodes in the multimodal hypergraph, and let H represent the association matrix of the multimodal hypergraph.

5. The classification method based on a multimodal brain hypergraph attention network with line graph extension according to claim 4, characterized in that, Step S6 specifically includes: The hypergraph attention network model consists of two stacked attention networks, each of which is divided into two layers based on the node layer and the hyperedge layer. The attention mechanism at the node layer is specifically represented as follows: In formula (4), W V σ represents the weight of the node obtained based on the updated hyperedge features at each layer. att Let HX represent a non-linear activation function. l-1 This represents the hypergraph node feature matrix updated from layer l-2, where, Let x represent the feature matrix of the M subjects obtained from the (l-1)th layer update. m Let H represent the multimodal features of the selected m-th subject, and let H represent the multimodal hypergraph correlation matrix. It is an attention vector; The specific representation of the attention mechanism of the super-edge layer is as follows: In formula (5), W E σ represents the weight of the hyperedge obtained based on the node features updated at each layer. att H represents a non-linear activation function. T X l-1 This represents the feature matrix of the line graph nodes obtained from the (l-2)th layer update, where, Let x represent the feature matrix of the M subjects obtained from the (l-1)th layer update. m H represents the brain region features of the selected m-th subject, and H represents the hypergraph association matrix. It is an attention vector; The specific steps for constructing a multimodal hypergraph attention network model using the line graph extension method are as follows: The node feature matrix of the line graph in layer l-1 comes from the updated hypergraph node features in this layer. The hyperedge features aggregated by the line graph in layer l-1 are redistributed to the nodes of the hypergraph and passed to the next layer. The feature information of the hypergraph nodes in layer (l-1) is mapped to the node feature matrix of the line graph in layer l in the form of matrix multiplication. Based on the attention mechanism of the hyperedge layer, the hyperedge weights are assigned to the line graph nodes. The line graph runs a hyperedge aggregation network to obtain hyperedge information and obtain the updated feature matrix. The specific formula is as follows: In formula (6), Let A represent the feature matrix of the (l-1)th layer obtained by aggregating hyperedge information. L This represents the weighted form of the adjacency matrix of a line graph, where D(·) denotes diagonal operation, and W... E H represents the weight of the hyperedge in the hypergraph obtained based on the node features updated at each layer. T X l-1 This represents the node feature matrix updated from layer l-2, where, Let x represent the feature matrix of the M subjects obtained from the (l-1)th layer update. m H represents the brain region features of the selected m-th subject, and H represents the hypergraph association matrix. It is a trainable parameter; The updated node features of the midline graph in layer (l-1) are mapped to the hypergraph in layer l. Node weights are learned based on the attention mechanism of the node layer. The node feature aggregation network obtains the node embedding by aggregating information from neighboring nodes according to the weights, and updates the feature matrix, as shown below: In formula (7), X l D represents the l-th layer feature matrix obtained by updating the aggregated node information, D(·) represents the diagonal operation, and W V H represents the weights of nodes in the hypergraph obtained based on the updated hyperedge features at each layer, and H represents the hypergraph association matrix. This represents the feature matrix of the (l-1)th layer obtained by aggregating hyperedge information. Q is a trainable parameter; The obtained supernode features are used to classify nodes through a softmax layer. Finally, the cross-entropy loss function is used on the training set to calculate the loss value, and stochastic gradient descent is used to optimize the cross-entropy error, thereby improving the model's classification ability. The specific expression of the softmax function is as follows: In formula (8), v represents the probability that the label of the i-th node is k. i This represents the i-th node, where e is the Napier constant; The cross-entropy loss function is expressed as follows: In formula (9), Represents hypergraph node v i The actual label is k. Represents hypergraph node v i The probability that the output label of the softmax function is the k-label.

6. The classification method based on a multimodal brain hypergraph attention network with line graph extension according to claim 1, characterized in that, In step S7, the cross-validation method is used to test the constructed multimodal hypergraph attention network. The specific steps are as follows: 90% of the samples in the dataset are randomly selected as the training set, and the remaining 10% are selected as the test set. Classification tests are performed to obtain the classification accuracy. The classification accuracy obtained after repeating cross-validation 100 times is arithmetically averaged, and then the arithmetic mean is used as the classification accuracy of the classifier.

Images