An intelligence prediction system based on magnetic resonance imaging and multi-scale multi-hop graph convolutional network

Abstract

This invention discloses an intelligence prediction system based on magnetic resonance imaging (fMRI) and a multi-scale, multi-hop graph convolutional network. The system takes functional magnetic resonance imaging (fMRI) data as input and outputs a predicted intelligence score for the experimental subject. It comprises three modules: a fMRI image processing module, a multi-scale brain region segmentation template loading module, and an intelligence prediction module. The fMRI image processing module preprocesses and saves the acquired fMRI images. The multi-scale brain region segmentation template loading module loads brain regions of interest at different scales according to requirements. The intelligence prediction module uses a multi-scale, multi-hop graph convolutional network to predict intelligence scores. This invention enables the prediction of intelligence.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of medical image analysis technology and computer science, and in particular to an intelligent prediction system based on magnetic resonance imaging and multi-scale multi-hop graph convolutional networks. Background Technology

[0002] Fluid intelligence refers to the human ability to solve new problems independently of previously acquired knowledge, playing a central role in learning, judgment, reasoning, and problem-solving. Identifying the neural mechanisms underlying fluid intelligence can advance our understanding of various complex cognitive tasks and cognitive development, and has therefore attracted widespread attention in recent years.

[0003] Resting-state functional magnetic resonance imaging (fMRI) provides a non-invasive and efficient technique for studying the relationship between fluid intelligence and brain tissue. By representing brain regions as nodes and the functional connectivity (FC) between nodes as edges, the brain is characterized as a complex network, also known as a functional connectivity network (FCN). Previous neuroscience findings have identified a correlation between intelligence and connectivity patterns within the FCN, particularly in the frontoparietal and default mode networks. Inspired by this, a growing body of research attempts to use features extracted from the FC to predict fluid intelligence scores.

[0004] In fact, numerous FCN-based fluid intelligence prediction models have been proposed using machine learning methods. These models rely on manual feature extraction and traditional regression construction. These features can be topological properties of the FC and FCN, as well as their derived features. Regression methods include linear regression, kernel ridge regression, and support vector regression (SVR). With the success of deep learning methods in various graph-related tasks, some deep neural networks (DNNs) have been applied to fluid intelligence prediction. Due to their amorphous graph structures, traditional DNNs cannot effectively extract feature representations from FCNs, while graph convolutional networks (GCNs) are considered a promising solution because they leverage graph topology to transfer node feature information, thus obtaining graph representation features. However, since fluid intelligence prediction using neuroimaging data is a very challenging task, existing GCN models fail to achieve satisfactory performance due to two limitations. First, the model learns features from a single-scale FCN, failing to depict the full picture of brain organization. Second, the graph convolutional layers only utilize latent information from nearest neighbors (1-hop neighbors), ignoring the rich information generated by long-distance functional interactions between brain regions. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an intelligence prediction system based on magnetic resonance imaging and multi-scale multi-hop graph convolutional networks, which can predict intelligence.

[0006] To address the aforementioned technical problems, this invention provides an intelligence prediction system based on magnetic resonance imaging (fMRI) and multi-scale multi-hop graph convolutional networks. The system takes functional magnetic resonance imaging (fMRI) data as input and outputs the predicted intelligence score of the experimental subjects. It mainly comprises three modules: a fMRI image processing module, a multi-scale brain region segmentation template loading module, and an intelligence prediction module. The fMRI image processing module preprocesses and saves the acquired fMRI images. The multi-scale brain region segmentation template loading module loads different scales of brain regions of interest according to requirements. The intelligence prediction module uses a multi-scale multi-hop graph convolutional network to predict intelligence scores.

[0007] Preferably, the system input is a four-dimensional functional magnetic resonance image of the experimental subject.

[0008] Preferably, the functional magnetic resonance imaging (fMRI) processing module preprocesses the fMRI data using head motion correction and filtering and smoothing preprocessing operations of the functional images.

[0009] Preferably, the multi-scale brain region segmentation template loading module is used to load brain regions of interest at different scales as needed.

[0010] Preferably, the intelligence prediction module includes a time series extraction module, a multi-scale brain functional network construction module, a feature extraction module, a multi-scale feature fusion module, and a fully connected module;

[0011] Time series of different voxels in functional magnetic resonance imaging of the brain are generated using a time series extraction module;

[0012] The multi-scale brain functional network construction module uses a set of brain segments from coarse to fine to generate brain functional connectivity networks of different scales. Specifically, it uses a set of maps with multiple resolutions and calculates the time-series Pearson correlation of blood oxygen level dependent signals between each pair of regions of interest based on the selected regions of interest.

[0013] The feature extraction module is based on the constructed multi-scale brain functional network. It uses multi-hop graph convolution, pooling and multilayer perceptron to extract functional connectivity feature representations at each scale, and adds inter-scale contrast constraints to ensure the similarity of functional connectivity network features across scales for experimental subjects.

[0014] The multi-scale feature fusion module introduces an attention method to fuse features at multiple scales after feature extraction.

[0015] Preferably, the feature extraction module, based on the constructed multi-scale brain functional network, uses multi-hop graph convolution, pooling, and multilayer perceptron to extract functional connectivity feature representations for each scale, and incorporates inter-scale contrast constraints to ensure the similarity of cross-scale functional connectivity network features among experimental subjects. Specifically, for each subject n, we represent the multi-scale functional connectivity network as a graph set G. n , M is the number of scales, where each graph It is the set of nodes at the m-th scale. It is an adjacency matrix. It is the characteristic matrix. The initial matrix is ​​the identity matrix; the specific steps include the following:

[0016] Step 1, Multi-hop graph convolution method: Update the feature representation of the current node by aggregating the feature representations of neighbors at different distances. Specifically, for a given graph... The features of the input nodes in the i-th layer are represented as follows: in It is the number of nodes, d i It is the node feature dimension of the i-th layer, and the output feature matrix of the i-th layer. The calculation is as follows:

[0017]

[0018] in It is a graph adjacency matrix The normalized adjacency matrix, where I is the identity matrix. yes The corresponding degree matrix, where k is the number of hops from a node to its neighbors. This represents the k-hop adjacency matrix. It is a parameter matrix, σ is the ReLU activation function, and || denotes column join;

[0019] Step 2, Pooling: After obtaining the feature representation of each graph output by the multi-hop graph convolution method, for the graph... We generate feature representations by pooling and aggregating node features. The specific method is as follows: for a given feature matrix of the nth subject at the mth scale... We use equation (2) to calculate the feature representation.

[0020]

[0021] in The feature representation of node v, It is the number of nodes;

[0022] Step 3, Multilayer Perceptron: Perform dimensionality reduction processing on the feature representation of the pooling output. For graphs... We use equation (3) to obtain the feature vector of the m-th scale for the n-th subject.

[0023]

[0024] Where Q represents the parameter matrix;

[0025] Step 4, Inter-scale contrast constraint: We designed the cross-scale contrast loss L inter To improve the similarity of cross-scale functional connectivity network features within the participants. For participant n, we consider feature representations from a multi-scale functional connectivity network within the same participant as positive pairs, and feature representations from different participants as negative pairs. We learn the correlation between the participants' multi-scale feature representations by minimizing the distance between positive pairs and maximizing the distance between negative pairs using equation (4):

[0026]

[0027] Here, Dist calculates the Euclidean distance between two vectors, and δ is the margin parameter. For positive alignment, let represent the feature vectors of scale m and scale m+1 of subject n. For negative pairs, it represents the feature vector of the same scale m for different subjects n and s.

[0028] Preferably, the multi-scale feature fusion module, after feature extraction, introduces an attention method to fuse features at multiple scales. Specifically, for subject n, let M scale features be represented as follows: We first calculate the weights of the m-th scale eigenvector. Right now:

[0029]

[0030] Where P m P m' σ is the parameter matrix, σ is the tanh activation function, and M is the scale number;

[0031] Then, we concatenate the multi-scale weighted features to generate the joint features z of subject n. n :

[0032]

[0033] The fully connected module integrates the joint features output by the multi-scale feature fusion module, specifically: the joint features z based on the nth subject. n Predicting intelligence scores for:

[0034]

[0035] Where U represents the parameter matrix. Finally, two supervised loss terms are used to predict the intelligence of all training objects, defined as:

[0036] L total =αL inter +L MAE (8)

[0037] The first term is the cross-scale contrastive loss defined in formula (4), and the second term is used to measure the accuracy of the prediction model:

[0038]

[0039] Where E represents the absolute error function, N is the number of training samples, and α is a hyperparameter used to balance the contributions of the two loss terms.

[0040] Preferably, the system outputs a predicted intelligence score for the experimental subjects.

[0041] The beneficial effects of this invention are as follows: This invention utilizes multi-scale multi-hop graph convolutional networks to learn the joint feature representation of multi-scale functional connectivity networks, and further designs inter-scale contrast constraints to improve the similarity of cross-scale functional connectivity network features within subjects; in addition, the multi-hop graph convolutional method learns the multi-hop neighbor information of nodes, improves the learning ability of traditional graph convolutional methods, can better capture brain functional connectivity information, and improves the intelligence prediction effect, which has important application value in the field of intelligence and other brain cognitive prediction. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the system structure of the present invention.

[0043] Figure 2 This is a schematic diagram of the intelligence prediction module of the present invention.

[0044] Figure 3 This is a schematic diagram of the experimental results of the present invention. Detailed Implementation

[0045] like Figure 1As shown, an intelligence prediction system based on magnetic resonance imaging (fMRI) and multi-scale multi-hop graph convolutional networks is presented. The system takes functional magnetic resonance imaging (fMRI) data as input and outputs predicted intelligence scores for experimental subjects. It mainly comprises three modules: a fMRI image processing module, a multi-scale brain region segmentation template loading module, and an intelligence prediction module. The fMRI image processing module preprocesses and saves the acquired fMRI images. The multi-scale brain region segmentation template loading module loads brain regions of interest at different scales according to requirements. The intelligence prediction module uses a multi-scale multi-hop graph convolutional network to predict intelligence scores.

[0046] The system takes as input four-dimensional functional magnetic resonance images of the experimental subjects.

[0047] The functional magnetic resonance imaging (fMRI) module preprocesses the fMRI data using head motion correction and filtering and smoothing preprocessing operations on the functional images.

[0048] The multi-scale brain region segmentation template loading module is used to load brain regions of interest at different scales according to requirements.

[0049] like Figure 2 As shown, the intelligence prediction module includes a time series extraction module, a multi-scale brain functional network construction module, a feature extraction module, a multi-scale feature fusion module, and a fully connected module.

[0050] Time series of different voxels in functional magnetic resonance imaging of the brain are generated using a time series extraction module;

[0051] The multi-scale brain functional network construction module uses a set of brain segments from coarse to fine to generate brain functional connectivity networks of different scales. Specifically, it uses a set of maps with multiple resolutions and calculates the time-series Pearson correlation of blood oxygen level dependent signals between each pair of regions of interest based on the selected regions of interest.

[0052] The feature extraction module, based on the constructed multi-scale brain functional network, utilizes multi-hop graph convolution, pooling, and multilayer perceptrons to extract functional connectivity feature representations for each scale. Inter-scale contrast constraints are added to ensure the similarity of cross-scale functional connectivity network features among experimental participants. For each participant n, the multi-scale functional connectivity network is represented as a graph set G. n , M is the number of scales. Each graph... It is the set of nodes at the m-th scale. It is an adjacency matrix. It is the characteristic matrix. The initial matrix is ​​the identity matrix; the specific steps include the following:

[0053] Step 1, Multi-hop graph convolution method: Update the feature representation of the current node by aggregating the feature representations of neighbors at different distances. Specifically, for a given graph... The features of the input nodes in the i-th layer are represented as follows: in It is the number of nodes, d i It is the node feature dimension of the i-th layer, and the output feature matrix of the i-th layer. The calculation is as follows:

[0054]

[0055] in It is a graph adjacency matrix The normalized adjacency matrix, where I is the identity matrix. yes The corresponding degree matrix, where k is the number of hops from a node to its neighbors. This represents the k-hop adjacency matrix. It is a parameter matrix, σ is the ReLU activation function, and || denotes column join;

[0056] Step 2, Pooling: After obtaining the feature representation of each graph output by the multi-hop graph convolution method, for the graph... We generate feature representations by pooling and aggregating node features. The specific method is as follows: for a given feature matrix of the nth subject at the mth scale... We use equation (2) to calculate the feature representation.

[0057]

[0058] in The feature representation of node v, It is the number of nodes;

[0059] Step 3, Multilayer Perceptron: Perform dimensionality reduction processing on the feature representation of the pooling output. For graphs... We use equation (3) to obtain the feature vector of the m-th scale for the n-th subject.

[0060]

[0061] Where Q represents the parameter matrix;

[0062] Step 4, Inter-scale contrast constraint: We designed the cross-scale contrast loss L interTo improve the similarity of cross-scale functional connectivity network features within the participants. For participant n, we consider feature representations from a multi-scale functional connectivity network within the same participant as positive pairs, and feature representations from different participants as negative pairs. We learn the correlation between the participants' multi-scale feature representations by minimizing the distance between positive pairs and maximizing the distance between negative pairs using equation (4):

[0063]

[0064] Here, Dist calculates the Euclidean distance between two vectors, and δ is the margin parameter. For positive alignment, let represent the feature vectors of scale m and scale m+1 of subject n. For negative pairs, it represents the feature vector of the same scale m for different subjects n and s;

[0065] The multi-scale feature fusion module introduces an attention method to fuse features at multiple scales after feature extraction. Specifically, for subject n, let M scale features be represented as follows: We first calculate the weights of the m-th scale eigenvector. Right now:

[0066]

[0067] Where P m P m' σ is the parameter matrix, σ is the tanh activation function, and M is the scale number;

[0068] Then, we concatenate the multi-scale weighted features to generate the joint features z of subject n. n :

[0069]

[0070] The fully connected module integrates the joint features output by the multi-scale feature fusion module, specifically: the joint features z based on the nth subject. n Predicting intelligence scores for:

[0071]

[0072] Where U represents the parameter matrix. Finally, two supervised loss terms are used to predict the intelligence of all training objects, defined as:

[0073] L total =αL inter +L MAE (8)

[0074] The first term is the cross-scale contrastive loss defined in formula (4), and the second term is used to measure the accuracy of the prediction model:

[0075]

[0076] Where E represents the absolute error function, N is the number of training samples, and α is a hyperparameter used to balance the contributions of the two loss terms.

[0077] The system outputs a predicted intelligence score for the experimental subjects.

[0078] This invention improves the learning ability of traditional graph convolution methods by learning joint feature representations of multi-scale functional connectivity networks through an intelligence prediction module. This allows for better capture of brain functional connectivity information and enhances intelligence prediction performance. The invention of this system contributes to improving the objective and accurate measurement of individual intelligence.

[0079] We used the Human Connectome Project (HCP) database, version S1200. The accuracy of intelligence prediction was evaluated by calculating the Pearson correlation between predicted and actual intelligence scores from subjects in the test set. We employed 5-fold cross-validation, repeated 5 times, to improve the robustness of the results. Therefore, we used the 25-fold average as the final prediction accuracy. Performance comparisons between our proposed method and four state-of-the-art intelligence prediction models (including Kernel Regression, Fully-connected Neural Network (FNN), BrainNetCNN, and Graph Convolutional Neural Network (GCNN) are shown below. Figure 3 As shown, our proposed method clearly demonstrates the best performance, with a Pearson correlation coefficient r = 0.262 ± 0.023, followed by FNN (0.169 ± 0.058), Kernel Regression (0.163 ± 0.049), BrainNetCNN (0.123 ± 0.060), and GCNN (0.054 ± 0.082). This validates the superiority of our proposed method, as it can extract rich information to generate robust and better representations.

Claims

1. An intelligence prediction system based on magnetic resonance imaging and multi-scale multi-hop graph convolutional networks, characterized in that, The system takes functional magnetic resonance imaging (fMRI) data as input and outputs predicted intelligence scores for experimental subjects. It comprises three modules: a fMRI image processing module, a multi-scale brain region segmentation template loading module, and an intelligence prediction module. The fMRI image processing module preprocesses and saves the acquired fMRI images. The multi-scale brain region segmentation template loading module loads brain regions of interest at different scales according to requirements. The intelligence prediction module uses a multi-scale, multi-hop graph convolutional network to predict intelligence scores. The intelligence prediction module includes a time-series extraction module, a multi-scale brain functional network construction module, a feature extraction module, a multi-scale feature fusion module, and a fully connected module. Time series of different voxels in functional magnetic resonance imaging of the brain are generated using a time series extraction module; The multi-scale brain functional network construction module uses a set of brain segments from coarse to fine to generate brain functional connectivity networks of different scales. Specifically, it uses a set of maps with multiple resolutions and calculates the time-series Pearson correlation of blood oxygen level dependent signals between each pair of regions of interest based on the selected regions of interest. The feature extraction module is based on the constructed multi-scale brain functional network. It uses multi-hop graph convolution, pooling and multilayer perceptron to extract the functional connectivity feature representations of each scale, and adds inter-scale contrast constraints to ensure the similarity of the functional connectivity network features of experimental subjects across scales. The multi-scale feature fusion module introduces an attention method to fuse features at multiple scales after feature extraction.

2. The intelligence prediction system based on magnetic resonance imaging and multi-scale multi-hop graph convolutional networks as described in claim 1, characterized in that, The system takes as input four-dimensional functional magnetic resonance images of the experimental subjects.

3. The intelligence prediction system based on magnetic resonance imaging and multi-scale multi-hop graph convolutional networks as described in claim 1, characterized in that, The functional magnetic resonance imaging (fMRI) module preprocesses the fMRI data using head motion correction and filtering and smoothing preprocessing operations on the functional images.

4. The intelligence prediction system based on magnetic resonance imaging and multi-scale multi-hop graph convolutional networks as described in claim 1, characterized in that, The multi-scale brain region segmentation template loading module is used to load brain regions of interest at different scales according to requirements.

5. The intelligence prediction system based on magnetic resonance imaging and multi-scale multi-hop graph convolutional networks as described in claim 1, characterized in that, The feature extraction module, based on the constructed multi-scale brain functional network, utilizes multi-hop graph convolution, pooling, and multilayer perceptrons to extract functional connectivity feature representations for each scale. It also incorporates inter-scale contrast constraints to ensure the similarity of cross-scale functional connectivity network features among experimental participants. Specifically, for each participant n, the multi-scale functional connectivity network is represented as a graph. M is the number of scales, where each graph It is the set of nodes at the m-th scale. It is an adjacency matrix. It is the characteristic matrix. The initial matrix is ​​the identity matrix; the specific steps include the following: Step 1, Multi-hop graph convolution method: Update the feature representation of the current node by aggregating the feature representations of neighbors at different distances. Specifically: for a given graph... The features of the input nodes in the i-th layer are represented as follows: in It is the number of nodes, d i It is the node feature dimension of the i-th layer, and the output feature matrix of the i-th layer. The calculation is as follows: in It is a graph adjacency matrix The normalized adjacency matrix, where I is the identity matrix. yes The corresponding degree matrix, where k is the number of hops from a node to its neighbors. Let W represent the k-hop adjacency matrix. i ∈ It is a parameter matrix, σ is the ReLU activation function, and || denotes column join; Step 2, Pooling: After obtaining the feature representation of each graph output by the multi-hop graph convolution method, for the graph... Feature representations are generated by pooling and aggregating node features. Specifically, for a given feature matrix at the m-th scale of the nth subject... The characteristic representation is calculated using equation (2). : in The feature representation of node v, It is the number of nodes; Step 3, Multilayer Perceptron: Perform dimensionality reduction processing on the feature representation of the pooling output. For graphs... The feature vector of the m-th scale of the n-th subject is obtained using equation (3). : Where Q represents the parameter matrix; Step 4, Inter-scale contrast constraint: A cross-scale contrast loss L was designed. inter To improve the similarity of cross-scale functional connectivity network features within the participants, for participant n, feature representations from a multi-scale functional connectivity network within one participant are considered positive pairs, while feature representations from different participants are considered negative pairs. The correlation between the participants' multi-scale feature representations is learned by minimizing the distance between positive pairs and maximizing the distance between negative pairs through equation (4). Here, Dist calculates the Euclidean distance between two vectors, and δ is the margin parameter. For positive alignment, let represent the feature vectors of scale m and scale m+1 of subject n. For negative pairs, it represents the feature vector of the same scale m for different subjects n and s.

6. The intelligence prediction system based on magnetic resonance imaging and multi-scale multi-hop graph convolutional networks as described in claim 1, characterized in that, The multi-scale feature fusion module, after feature extraction, introduces an attention method to fuse features at multiple scales. Specifically, for subject n, let M scale features be represented as follows: First, calculate the weights of the m-th scale eigenvector. Right now: Where P m P m′ σ is the parameter matrix, σ is the tanh activation function, and M is the scale number; Then, the multi-scale weighted features are concatenated to generate the joint features z of subject n. n : The fully connected module integrates the joint features output by the multi-scale feature fusion module, specifically: the joint features z based on the nth subject. n Predicting intelligence scores for: Where U represents the parameter matrix, and finally, two supervised loss terms are used to predict the intelligence of all training subjects, defined as: L total =αL inter +L MAE (8) The first term is the cross-scale contrastive loss defined in formula (4), and the second term is used to measure the accuracy of the prediction model: Where E represents the absolute error function, N is the number of training samples, and α is a hyperparameter used to balance the contributions of the two loss terms.

7. The intelligence prediction system based on magnetic resonance imaging and multi-scale multi-hop graph convolutional networks as described in claim 1, characterized in that, The system outputs a predicted intelligence score for the experimental subjects.

Images