Autism identification method based on multi-view image pooling

Abstract

The invention discloses an autism identification method based on multi-view image pooling, which relates to the technical field of biomedical information and comprises the following steps: starting from two aspects of local feature extraction and global feature extraction, aiming at local feature extraction, constructing an individual brain image of each subject, extracting brain features according to an image pooling strategy, and extracting an individual brain image of each subject according to the image pooling strategy; sorting the nodes from multiple angles of structures and features, and screening to obtain sparse features; and for global feature extraction, summarizing the features of each main body into a row of vectors, obtaining the overall graph-level representation of a coarsened graph, and automatically learning the statistical features of functional nuclear magnetic resonance image data by using a multi-layer perceptron to obtain high-order abstract feature representation. And the image neural network model further fuses the imaging data and the non-imaging data. And constructing a population map by taking the high-order abstract feature of each subject as a node feature and taking the similarity of the population statistical information as an edge weight, training a map neural network, and outputting an autism recognition result.

Description

Technical Field

[0001] The present invention relates to the field of biomedical information technology, and in particular to an autism recognition method based on multi-view graph pooling. Background Art

[0002] Autism is a major developmental disorder characterized by social communication and interaction disorders, sensory abnormalities, repetitive behaviors, and varying degrees of intellectual disability, often co-existing with other diseases. The global prevalence of autism is slightly less than 1%, but it may be much higher in high-income countries. Most people with autism cannot live independently, which is also a major economic burden on families and society. However, the diagnosis of autism is largely based on simple symptom observation and clinician experience, and there is a lack of reliable biomarkers. A lot of work is still needed to understand its underlying mechanisms and develop effective treatments.

[0003] Resting-state fMRI (rs-fMRI) mainly reflects changes in blood oxygen levels. It can capture interactions between brain regions and is expected to reveal abnormal connections in the human brain. Many methods tend to construct functional connectivity networks after extracting time series signals from the original images. In general, functional connectivity networks are constructed using connectivity metrics such as Pearson correlation coefficients or partial correlation coefficients to capture the interactions between each two regions. However, functional connectivity networks are prone to produce densely connected brain networks, leading to overfitting problems and increasing computational complexity. In addition, most deep learning models focus more on grid structure data, while brain network data do not follow the spatial distribution of regions. Summary of the invention

[0004] The purpose of the present invention is to provide an autism recognition method based on multi-view graph pooling to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for autism recognition based on multi-view graph pooling includes local feature extraction and global feature extraction. Local features refer to the brain area features of each subject. The specific steps are as follows: for local feature extraction, construct an individual brain map of each subject, extract brain features according to the graph pooling strategy, sort the nodes from multiple perspectives of structure and features, and screen out sparse features; for global feature extraction, summarize the features of each subject into a row vector to obtain the overall graph-level representation of the coarsened graph, use a multi-layer perceptron to automatically learn the statistical features of functional magnetic resonance imaging data, and obtain a high-order abstract feature representation. The graph neural network model further integrates imaging data and non-imaging data, uses the high-order abstract features of each subject as node features, and uses the similarity of demographic information as edge weights to construct a population map, train the graph neural network, and output autism recognition results.

[0007] Furthermore, for local feature extraction, the following steps are specifically included:

[0008] Construct an individual brain map of a single subject. The brain region features extracted from the Harvard-Oxford brain partitioning atlas include 55 brain regions from each hemisphere. Add the global average time series. The final constructed individual brain map contains 111 nodes. Connect the nodes of the left brain region to the right, and the nodes of the right brain region to the left to construct an individual brain map for each subject.

[0009] The local brain region features are extracted from three view levels respectively. The brain region nodes in multiple views are sorted by using the context graph information. The robust sorting results are conducive to extracting effective brain region features of different subjects. For the structural view, the representation of the learned graph structure is derived from the topological structure of the graph. The degree centrality is selected as the structural view score to generate x s , the formula is as follows:

[0010] x s =σ(α*log(D l +ε)+β);

[0011] Among them, D l represents the degree matrix of layer l, ε is a small positive constant to avoid logarithmic operation on 0, and an activation function σ(·) is used to map these values ​​to the range [0,1] to match the scale in other views; α and β are used to control the scaling and shifting of the degree values;

[0012] For the feature view, the time series of brain regions are represented as nodes, and a multi-layer perceptron (MLP) is used to transform the features of the nodes into high-level abstract features as the score generator x f , the formula is as follows:

[0013] x f =σ(MLP(Hl ));

[0014] Among them, H l The node representation matrix representing the subject in the lth layer, MLP(·) is also called a fully connected layer, that is, the neurons between layers are interconnected; the σ function normalizes the feature scores obtained through MLP;

[0015] For the combined view of structure and features, a graph neural network is used to combine structural information and feature information, extract graph information, and generate multi-view score generator x sf The formula is as follows:

[0016] x sf =σ(γ(I n -(1-γ)A l ) -1 H l );

[0017] Among them, γ∈(0,1) is the transmission (or restart) probability, I n Represents the identity matrix. First, the graph structure and node feature information (A and H) are mapped to [0, 1] through a graph neural network. In addition, the value is approximated by power iteration to reduce the computational complexity.

[0018] Combining the scores of multiple views, the results of the above score generator are weighted and expressed as:

[0019]

[0020] Where x is the final score obtained by merging all views, x c Score the score generator for a view, p c is the corresponding weight, and the calculation formula is as follows:

[0021]

[0022] where v is composed of different fractions x c In the above views, w and b are learnable parameters of view c, and the weights are normalized using the softmax function;

[0023] Finally, a node selector is defined to select nodes with significant scores in the top-k, and the selected nodes are used to coarsen the original graph to obtain a new individual brain map. The formula is as follows:

[0024]

[0025] A l+1 =A l (idx,idx)

[0026] Where r is the pooling rate, the TOPK function controls the number of nodes retained in each layer according to r, x) is the node score after graph pooling; the sigmoid activation function is used to perform nonlinear transformation of node scores, H l+1 and A l+1 They correspond to the representation matrix and adjacency matrix after graph pooling respectively.

[0027] Furthermore, for global feature extraction, the following steps are specifically included:

[0028] The features of each subject are summarized into a row vector through the readout layer to obtain the overall graph-level representation of the coarsened graph, and the generated pooled features are Input into MLP(·) to extract more discriminative high-order features, the formula is as follows:

[0029]

[0030] in By X max and X avg Splicing, X max Take the maximum value of the pooled feature on the node as the graph-level feature, X avg Take the average of the topic node features as the graph-level feature, x i represents the feature of the i-th brain region, and N is the number of nodes / brain regions obtained after pooling;

[0031] Construct the overall crowd graph, integrate the image data and non-image data, and define the edge weights as follows:

[0032]

[0033] Among them, sim(X i ,X j ) is the distance of image data, the latter half is the distance of non-image data, M represents different types of non-image data, for category data, their values ​​are calculated by Kronecker delta function, and for numerical data, their similarity is obtained by setting a threshold, and then C′ is binarized to obtain a binary connection graph C, and the crowd graph a is as follows:

[0034] A=CeS

[0035]

[0036] where p is the projection of the phenotypic data normalized into a common latent space by the MLP, S ij is the adjusted cosine similarity between subject i and subject j. The similarity between A can be optimized by the subsequent graph neural network model parameters and minimizing the loss function;

[0037] The image data and non-image data obtained above are input into the graph neural network to extract effective discriminant features. The construction formula of the graph neural network is as follows:

[0038] x*g=U((U T x)·(U T g))=Ug(Λ)U T x;

[0039] Where Λ is the eigenvalue matrix of the Laplace matrix. The Laplace matrix is ​​decomposed and the Chebyshev polynomial is introduced to approximate the convolution kernel:

[0040]

[0041] where θ k is a learnable parameter, Tn(x) is an n-order Chebyshev polynomial, and the final expression of the simplified graph convolution layer is:

[0042]

[0043] Finally, a skip connection mechanism was introduced to aggregate domain information of different nodes to different degrees. The Monte Carlo dropout method was added to obtain the prediction results and uncertainties at the same time. The model was iteratively trained through the binary cross entropy loss function to obtain the optimal recognition model to classify autism data.

[0044] Compared with the prior art, the present invention has the following advantages: (1) when analyzing functional magnetic resonance imaging data, different from the common methods, the model of the present invention extracts brain features through a graph pooling strategy, and sorts nodes from multiple perspectives of structure and features by utilizing contextual graph information. The robust sorting results are conducive to extracting effective brain area features of different subjects and help alleviate the over-smoothing problem.

[0045] (2) By performing personalized brain region selection for each subject and analyzing the associations between subjects in a population graph, more potential information can be obtained by combining local and global aspects, which promotes the research on effective biomarkers.

[0046] (3) Introducing graph neural networks to fuse imaging data and non-imaging data into autism diagnosis, and using graph networks to dynamically update population graphs and dynamically adjust edge weights between subjects can improve autism diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of a flow chart of an embodiment of the present invention;

[0048] Figure 2 A method framework diagram of an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of a graph pooling method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] like Figure 1 The flowchart of the autism recognition method based on multi-view graph pooling in this example is shown, which includes the following steps:

[0052] In step S1, the ABIDE dataset used is a functional MRI data collected from 20 sites. The data preprocessing uses the configurable Pipeline for Analysis of Connectomes (C-PAC), whose main steps include slice timing correction, motion correction, skull striping, global mean intensity normalization, interference signal regression, and bandpass filtering (0.01-0.1Hz). The functional image is aligned to the anatomical space by linear transformation, and the generated map is normalized to the standard anatomical space. The preprocessed fMRI data is segmented into 110 brain regions using the Harvard-Oxford (HO) template, which is divided into cortical and subcortical probability maps. The left / right WM, left / right GM, left / right CSF, and brainstem are removed from the subcortical map. The HO map is divided into left and right cerebral hemispheres at the midline using a 25% threshold. Finally, 871 subjects were used after quality inspection, including 403 ASD patients and 468 typical controls. The non-imaging data selected the gender, age, and site of data collection corresponding to the subject.

[0053] In step S2, an individual brain map of a single subject is constructed. The brain region features extracted by the Harvard-Oxford brain partitioning atlas include 55 brain regions from each hemisphere. In addition, in order to further reduce the error, the global average time series is added, and the final constructed individual brain map contains 111 nodes. The nodes of the left brain region are connected to the right side, and the nodes of the right brain region are connected to the left side. The individual brain map of each subject is constructed, and each brain map includes 6270 edges.

[0054] In step S3, the data of 871 subjects obtained in step S1 are divided into a training set and a test set by ten-fold cross-validation, wherein each fold of the training set is again divided by ten-fold cross-validation, with nine folds as the training set and one fold as the validation set, for model training.

[0055] In step S4, the processed image data and the constructed individual brain map are input to extract local features, and local brain area features are extracted from three view levels respectively. By using the context map information to sort the brain area nodes in multiple views, the robust sorting result is conducive to extracting effective brain area features of different subjects. Specifically, the following steps are included:

[0056] A. For the structural view, the representation of the learned graph structure only comes from the topological structure of the graph, and the degree centrality is selected as the structural view score to generate x s , the formula is as follows:

[0057] x s =σ(α*log(D l +ε)+β)

[0058] Where Dl represents the degree matrix of the lth layer, and ε is a small positive constant to avoid logarithmic operation on 0. These values ​​are mapped to the range [0,1] by an activation function σ(·) to facilitate the scale matching in other views. α and β are used to control the scaling and shifting of the degree values.

[0059] B. For the feature view, the time series of brain regions are represented as nodes. We use a multi-layer perceptron (MLP) to transform the features of the nodes into high-level abstract features as the score generator x f , the formula is as follows:

[0060] x f =σ(MLP(H l ))

[0061] Among them, H l The nodes representing the subjects in the lth layer represent the matrix. MLP(·) is also called a fully connected layer, that is, the neurons between layers are interconnected. Similarly, the σ function normalizes the feature scores obtained through MLP.

[0062] C. For the combined view of structure and features, in order to obtain a more comprehensive representation, a graph neural network is used to combine the structural information and feature information to extract graph information. Multi-view score generator x sf The formula is as follows:

[0063] x sf =σ(γ(I n -(1-γ)A l ) -1 H l )

[0064] Among them, γ∈(0,1) is the transmission (or restart) probability, I nRepresents the identity matrix. First, the graph structure and node feature information (A and H) are mapped to [0,1] through a graph neural network. In addition, the value is approximated by power iteration to reduce the computational complexity.

[0065] D. Combining the scores of multiple views, weight the results of the above score generators, expressed as:

[0066]

[0067] Where x is the final score obtained by merging all views, x c Score the score generator for a view, p c is the corresponding weight, and the calculation formula is as follows:

[0068]

[0069] where v is composed of different fractions x c In the above views, w and b are learnable parameters of view c, and the weights are normalized using the softmax function.

[0070] E. A node selector is defined to select nodes with significant scores in the top-k, and the selected nodes are used to coarsen the original graph to obtain a new individual brain map. The formula is as follows:

[0071]

[0072] A l+1 =A l (idx,idx)

[0073] Where r is the pooling rate, the TOPK function controls the number of nodes retained in each layer according to r, and x) is the node score after graph pooling. The sigmoid activation function is used to perform nonlinear transformation of node scores. and A l+1 They correspond to the representation matrix and adjacency matrix after graph pooling respectively.

[0074] In step S5, the overall crowd graph is constructed by fusing the image data and non-image data, and the edge weights are defined as follows:

[0075]

[0076] Among them, sim(X i ,X j) is the distance of image data, the latter half is the distance of non-image data, and M represents different types of non-image data. For categorical data, their values ​​are directly calculated by Kronecker delta function, and for numerical data, their similarity is obtained by setting a threshold. Then C′ is binarized to obtain the binary connection graph C, and the crowd graph a is as follows:

[0077] A=CeS

[0078]

[0079] where p is the projection of the phenotypic data normalized into a common latent space by the MLP, S ij is the adjusted cosine similarity between subject i and subject j. The similarity between A can be optimized by the subsequent graph neural network model parameters and minimizing the loss function.

[0080] In step S6, the image data and non-image data obtained in steps S4 and S5 are input into the graph neural network to extract effective discriminant features. The construction formula of the graph neural network is as follows:

[0081] x*g=U((U T x)·(U T g))=Ug(Λ)U T x

[0082] Where Λ is the eigenvalue matrix of the Laplace matrix. The Laplace matrix is ​​decomposed and the Chebyshev polynomial is introduced to approximate the convolution kernel:

[0083]

[0084] where θ k is a learnable parameter, Tn(x) is an n-order Chebyshev polynomial, and the final expression of the simplified graph convolution layer is:

[0085]

[0086] Finally, a skip connection mechanism is introduced to aggregate domain information of different nodes to different degrees, and the Monte Carlo dropout method is added to obtain the prediction results and uncertainty at the same time.

[0087] In step S7, the model is iteratively trained using a binary cross entropy loss function to obtain an optimal recognition model for classifying and predicting autism data.

[0088] In step S8, the model results of step 7 are evaluated, and the model performance is compared from four aspects. Accuracy reflects the ability of the model to predict the correct samples, while sensitivity can measure the degree of recognition of the disease. Specificity refers to the chance of not misdiagnosing the disease. The F1 value is the harmonic mean of the recall rate and the precision rate, so that both are maximized at the same time and balanced. The calculation formulas are as follows:

[0089]

[0090] Among them, TP is the number of samples correctly predicted as positive examples, TN is the number of samples correctly predicted as duplicate examples, FP is the number of samples incorrectly predicted as positive examples, and FN is the number of samples incorrectly predicted as negative examples.

[0091] The present invention is further described with reference to an embodiment:

[0092] The entire framework is built on PyTorch using NVIDIA TITAN RTX GPUs, trained using the above steps, and compared with eight methods in two categories:

[0093] The first category is the traditional machine learning model:

[0094] The functional magnetic resonance imaging data extracted in step S1 are used to construct brain network connection features through the Pearson correlation coefficient, and the upper triangular vectorization is taken as the feature of each subject, which is used as the input of the support vector machine and the ridge classification method. The obtained result is a reference benchmark.

[0095] The second category is the model based on graph neural network:

[0096] PopulationGCN: For the first time, GCN was creatively applied to the diagnosis of ASD, constructing a population graph and achieving groundbreaking experimental results.

[0097] MVS-GCN: Using multi-view data, we introduce prior knowledge and use the known brain network structure to guide the learning process of the graph convolutional network, which combines graph structure learning and multi-task graph embedding learning to learn effective representations of brain networks in an end-to-end manner.

[0098] BrainGNN: A novel ROI-aware graph convolutional layer is designed. The ROI selection pooling layer can highlight significant ROIs by utilizing the topological and functional information of fMRI. Regularization terms - unit loss, topK pooling loss and group-level consistency loss - are introduced on the pooling results to improve the rationality of ROI selection.

[0099] EV-GCN: We propose a novel learnable adaptive variational edge population graph framework that can automatically integrate imaging and non-imaging data in the population for uncertainty-aware disease prediction.

[0100] LG-GNN: Two feature extraction modules are designed, ROI-GNN learns feature embeddings of local brain regions and identifies biomarkers, and Subject-GNN learns the relationship between subjects and the embeddings generated by local ROI-GNN and non-imaging information.

[0101] GP-LC: A new fMRI feature extraction method is proposed to learn personalized low-resolution representations of the entire brain network regarding functional connectivity and regional activity, embedding sparse feature vectors into population graphs and using graph convolutional networks to learn node embeddings.

[0102] The experimental results of the present invention and the above comparative methods are shown in Table 1.

[0103] Table 1 Performance comparison of different methods on the ABIDE dataset

[0104]

[0105] From the comparison of evaluation indicators in Table 1, it can be found that the autism recognition method based on multi-view graph pooling has significant advantages.

[0106] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0107] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for autism recognition based on multi-view graph pooling, characterized in that: It includes local feature extraction and global feature extraction. Local features refer to the brain region features of each subject. The specific steps are as follows: for local feature extraction, construct an individual brain map of each subject, extract brain features according to the graph pooling strategy, sort the nodes from multiple perspectives of structure and features, and screen out sparse features; For global feature extraction, the features of each subject are summarized into a row vector to obtain the overall graph-level representation of the coarsened graph. The multilayer perceptron is used to automatically learn the statistical features of the functional magnetic resonance imaging data to obtain a high-order abstract feature representation. The graph neural network model further integrates imaging data and non-imaging data. The high-order abstract features of each subject are used as node features, and the similarity of demographic information is used as the edge weight to construct a population graph. The graph neural network is trained to output autism recognition results.

2. The autism recognition method based on multi-view graph pooling according to claim 1, characterized in that: For local feature extraction, the following steps are specifically included: Construct an individual brain map of a single subject. The brain region features extracted from the Harvard-Oxford brain partitioning atlas include 55 brain regions from each hemisphere. Add the global average time series. The final constructed individual brain map contains 111 nodes. Connect the nodes of the left brain region to the right, and the nodes of the right brain region to the left to construct an individual brain map for each subject. The local brain region features are extracted from three view levels respectively. The brain region nodes in multiple views are sorted by using the context graph information. The robust sorting results are conducive to extracting effective brain region features of different subjects. For the structural view, the representation of the learned graph structure is derived from the topological structure of the graph. The degree centrality is selected as the structural view score to generate x s , the formula is as follows: x s =σ(α*log(D l +e)+b); Among them, D l represents the degree matrix of layer l, ε is a small positive constant to avoid logarithmic operation on 0, and an activation function σ(·) is used to map these values ​​to the range [0,1] to match the scale in other views; α and β are used to control the scaling and shifting of the degree values; For the feature view, the time series of brain regions are represented as nodes, and a multi-layer perceptron (MLP) is used to transform the features of the nodes into high-level abstract features as the score generator x f , the formula is as follows: x f =σ(MLP(H l )); Among them, H l The node representation matrix representing the subject in the lth layer, MLP(·) is also called a fully connected layer, that is, the neurons between layers are interconnected; the σ function normalizes the feature scores obtained through MLP; For the combined view of structure and features, a graph neural network is used to combine structural information and feature information, extract graph information, and generate multi-view score generator x sf The formula is as follows: x sf =σ(γ(I n -(1-c)A l ) -1 H l ); Among them, γ∈(0,1) is the transmission (or restart) probability, I n Represents the identity matrix. First, the graph structure and node feature information (A and H) are mapped to [0, 1] through a graph neural network. In addition, the value is approximated by power iteration to reduce the computational complexity. Combining the scores of multiple views, the results of the above score generator are weighted and expressed as: Where x is the final score obtained by merging all views, x c Score the score generator for a view, p c is the corresponding weight, and the calculation formula is as follows: where v is composed of different fractions x c In the above views, w and b are learnable parameters of view c, and the weights are normalized using the softmax function; Finally, a node selector is defined to select nodes with significant scores in the top-k, and the selected nodes are used to coarsen the original graph to obtain a new individual brain map. The formula is as follows: A l+1 =A l (idx,idx) Where r is the pooling rate, the TOPK function controls the number of nodes retained in each layer according to r, x) is the node score after graph pooling; the sigmoid activation function is used to perform nonlinear transformation of node scores, and A l+1 They correspond to the representation matrix and adjacency matrix after graph pooling respectively.

3. The autism recognition method based on multi-view graph pooling according to claim 1, characterized in that: For global feature extraction, the following steps are specifically included: The features of each subject are summarized into a row vector through the readout layer to obtain the overall graph-level representation of the coarsened graph, and the generated pooled features are Input into MLP(·) to extract more discriminative high-order features, the formula is as follows: in By X max and X avg Splicing, X max Take the maximum value of the pooled feature on the node as the graph-level feature, X avg Take the average of the topic node features as the graph-level feature, x i represents the feature of the i-th brain region, and N is the number of nodes / brain regions obtained after pooling; Construct the overall crowd graph, integrate the image data and non-image data, and define the edge weights as follows: Among them, sim(X i ,X j ) is the distance of image data, the latter half is the distance of non-image data, M represents different types of non-image data, for category data, their values ​​are calculated by Kronecker delta function, and for numerical data, their similarity is obtained by setting a threshold, and then C′ is binarized to obtain a binary connection graph C, and the crowd graph a is as follows: A=CeS where p is the projection of the phenotypic data normalized into a common latent space by the MLP, S ij is the adjusted cosine similarity between subject i and subject j. The similarity between A can be optimized by the subsequent graph neural network model parameters and minimizing the loss function; The image data and non-image data obtained above are input into the graph neural network to extract effective discriminant features. The construction formula of the graph neural network is as follows: x*g=U((U T x)·(U T g))=Ug(Λ)U T x; Where Λ is the eigenvalue matrix of the Laplace matrix. The Laplace matrix is ​​decomposed and the Chebyshev polynomial is introduced to approximate the convolution kernel: where θ k is a learnable parameter, Tn(x) is an n-order Chebyshev polynomial, and the final expression of the simplified graph convolution layer is: Finally, a skip connection mechanism was introduced to aggregate domain information of different nodes to different degrees. The Monte Carlo dropout method was added to obtain the prediction results and uncertainties at the same time. The model was iteratively trained through the binary cross entropy loss function to obtain the optimal recognition model to classify autism data.

Images