Multimodal medical image prediction method and device based on graph neural network, medium and product

Abstract

The application discloses a kind of multi-modal medical image prediction method, equipment, medium and product based on graph neural network, belong to medical image processing technical field.The application is first to each modality MRI image data preprocessing, obtain the graph data of each modality, then based on the feature extraction and fusion module of multi-modal brain age prediction model extracts the feature of the graph data of each modality, again fusion obtains multi-modal fusion feature vector, again the multi-modal fusion feature vector is input to the prediction module of multi-modal brain age prediction model, to obtain brain age prediction result, the loss function when training prediction model includes prediction loss and the reconstruction loss of each modality graph data, i.e. The application fully utilizes the MRI image information of multiple modalities of the subject, learns nonlinear features based on the GNN autoencoder model, and effectively improves the model performance.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and in particular relates to a multimodal medical image prediction method, device and medium based on graph neural networks. Background Technology

[0002] Mental illness severely impacts the quality of life for patients and their families, and the patients' extreme behaviors can even endanger public safety. Currently, most mental illness assessments rely on doctors' subjective judgments, lacking effective objective methods. In recent years, with the rapid development of artificial intelligence and medical imaging technology, neuroimaging-based brain age prediction methods have advanced rapidly. By comparing the degree of deviation from the normal trajectory of brain aging, these methods assess the impact of mental illness, providing a reliable method for objective evaluation, and have shown significant effectiveness in the fields of Alzheimer's disease, Parkinson's disease, and other mental illnesses. Simultaneously, multimodal learning improves model performance by learning complementary information from data from multiple modalities. Furthermore, functional connectivity extracted from resting-state magnetic resonance imaging (MRI) is often modeled as graphical data representing the correlations between different brain regions. Therefore, multimodal medical image prediction methods based on graph neural networks (GNNs) are of significant research importance.

[0003] In recent years, medical image-assisted prediction methods have developed rapidly, from machine learning methods to a large number of deep learning methods, and from single-modal input to multimodal learning, applied to various mental illnesses. Currently, most brain age prediction methods can be divided into two main categories: one is machine learning methods that use extracted one-dimensional features, such as the gray matter volume of various brain regions, as one-dimensional input to traditional machine learning models; the other is deep learning methods that input 2D or 3D raw images or preprocessed gray matter images into corresponding 2D or 3D neural networks.

[0004] The paper "Graph Transformer Geometric Learning of Brain Networks Using Multimodal MR Images for Brain Age Estimation" (H.Cai, Y.Gao, and M.Liu, in IEEE Transactions on Medical Imaging, vol.42, no.2, pp.456-466, Feb. 2023) achieved high-performance multimodal brain age prediction using GNNs, with a mean absolute error (MAE) as low as 2.71 years. However, the paper did not utilize resting-state functional MRI data, which can detect brain changes caused by mental illness earlier than structural MRI data. In the literature "Predicting brain age from functional connectivity insymptomatic and preclinical Alzheimer disease" [J]. Millar PR, Luckett PH, Gordon BA, et al., NeuroImage, 2022, 256:119228, the traditional machine learning method Gaussian process regression based on resting-state functional MRI data achieved a MAE of 8.20. The experimental results show that resting-state MRI data is suitable for predicting brain age and revealing accelerated brain aging in Alzheimer's patients. However, single-modality feature learning and traditional machine learning methods limit the improvement of model performance.

[0005] Deep learning-based medical image-assisted prediction methods offer the possibility of objectively assessing changes in brain structure and function in patients with mental illnesses. However, this method still has limitations; single-modal prediction models cannot fully extract and utilize all available information from the subjects, thus hindering high performance.

[0006] In patent application CN115187513A, a network model Tri-Unet based on 3D ResNet and 3D UNet was proposed to better learn the features of deep and shallow layers, as well as front and back layers. In patent application CN110859624A, an efficient and accurate brain age prediction model based on Age-DenseNet was proposed, with input being complex 3D structural MRI data that had undergone image registration, skull dissection, and image data standardization, thus solving the problem of low feature utilization efficiency. Both of these methods use single-modality data, therefore, model performance needs improvement. In patent application CN113616184A, the inventors solved the problem of inconsistent image information data formats obtained from different modal preprocessing, but this invention uses traditional machine learning methods rather than deep learning methods more suitable for learning nonlinear features. Summary of the Invention

[0007] This invention provides a multimodal medical image prediction method, device, and medium based on graph neural networks to address the technical problem of the lack of objective assessment methods for mental illnesses.

[0008] On the one hand, the present invention provides a multimodal medical image prediction method based on graph neural networks, the method comprising the following steps:

[0009] Step 1: Perform data preprocessing on the MRI images of various modalities acquired by the input magnetic resonance imaging instrument, and then construct graph data of each modal MRI image data after preprocessing to obtain graph data of each modality.

[0010] Step 2: Construct a multimodal brain age prediction model (AgeGNN) based on graph neural networks. This prediction model includes a feature extraction and fusion module and a prediction module.

[0011] The feature extraction and fusion module includes several feature extraction branches, the number of which is consistent with the number of modalities of the MRI images acquired in step 1. Each feature extraction branch includes a set of encoders and decoders. The encoders and decoders have the same network structure, each including several unit modules composed of graph transform convolution (TransformerConv), batch normalization layers (BatchNorm), and activation functions (ReLU). The encoder's input is single-modality graph data, which extracts features from the graph data based on downsampling. The decoder's input is the encoder's output, which reconstructs the single-modality graph data based on upsampling. All feature extraction branches have the same number of unit modules in their autoencoders, and the extracted features have the same dimension.

[0012] The feature extraction and fusion module fuses the feature vectors output by the encoders of each feature extraction branch through the splicing layer to obtain a multi-mode fused feature vector, which is then input into the prediction module.

[0013] The prediction module performs a nonlinear transformation on the multi-modal fusion feature vector using a multilayer perceptron (MLP) to obtain the predicted brain age.

[0014] Step 3: Optimize the model parameters of the multimodal brain age prediction model based on the training data. When the preset training termination condition is met, obtain the age predictor for multimodal medical images based on the autoencoder of each feature extraction branch that has been trained, as well as the splicing layer and prediction module.

[0015] The loss function during optimized training includes the reconstruction loss between the encoder input and the decoder reconstructed unimodal graph data for each feature extraction branch, and the prediction loss between the predicted brain age and the label output by the prediction module.

[0016] Furthermore, the loss function during optimization training is set as follows:

[0017]

[0018] Where L represents the loss function during optimization training, L pred L represents the predicted loss. recon,j Let represent the reconstruction loss, j represent the modality identifier, N be the number of samples, M be the number of modalities, μ be the regularization coefficient, w be the model parameters of the multimodal brain age prediction model, and i represent the sample index. This indicates that the decoder reconstructs the single-modal graph data, x i,j BA represents the input of the autoencoder. i Indicates the prediction of brain age, CA i Indicates the actual age.

[0019] That is, the loss function L of this invention includes the prediction loss L using the mean square error (MSE). pred Reconstruction loss L from multiple modal data recon,j At the same time, an L2 regularization term is added to prevent overfitting and improve generalization.

[0020] Furthermore, in step 1, data preprocessing specifically includes:

[0021] Brain regions were processed separately for each modality of MRI images based on the same brain region template;

[0022] Then, each brain region is used as a node in the graph structure, the connections between brain regions are used as edges in the graph structure, and the vector composed of the correlation coefficients (e.g., Pearson correlations) between the features of the corresponding brain region and all other brain regions is used as the node features. The graph structure data of MRI image data for each modality is constructed based on the correlation coefficients between the features of the corresponding brain regions as the edge features, thus obtaining the graph data for each modality.

[0023] Furthermore, the prediction module consists of a linear layer, an activation function (ReLU), a batch normalization layer (BatchNorm), and another linear layer.

[0024] On the other hand, the present invention also discloses a computing device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the multimodal medical image prediction method based on graph neural networks of the present invention.

[0025] On the other hand, the present invention also discloses a computer-readable storage medium storing a computer program / instruction thereon, characterized in that the computer program / instruction, when executed by a processor, implements the multimodal medical image prediction method based on graph neural networks of the present invention.

[0026] In another aspect, the present invention also discloses a computer program product, including a computer program / instruction, characterized in that the computer program / instruction, when executed by a processor, implements the multimodal medical image prediction method based on graph neural networks of the present invention.

[0027] The technical solution provided by this invention brings at least the following beneficial effects:

[0028] This invention realizes the fusion of multimodal medical image information, promoting the application of multimodal technology in the field of medical imaging;

[0029] This invention uses a GNN-based autoencoder model to extract features, which reduces the evaluation metrics for regression tasks such as MAE.

[0030] This invention constructs a multimodal medical image prediction method, which makes full use of the MRI image information of the subject in multiple modalities, and learns nonlinear features through a GNN-based autoencoder model, effectively improving the model performance. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1This is a block diagram of the multimodal medical image prediction method based on graph neural networks according to an embodiment of the present invention;

[0033] Figure 2 This is a structural diagram of AgeGNN, a multimodal brain age prediction model based on graph neural networks, according to an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0035] The dataset used in this embodiment of the invention comes from the Alzheimer's Disease Neuroimaging Initiative (ADNI). ADNI is a multi-site longitudinal data repository containing a large amount of T1-weighted structural MRI, functional MRI, diffusion tensor imaging (DTI) and other MRI data.

[0036] See Figure 1 and Figure 2 The present invention provides a multimodal medical image prediction method based on graph neural networks, comprising the following steps:

[0037] Step S1: Perform MRI image data preprocessing on the original T1-weighted structural MRI, resting functional MRI, and DTI data to obtain three sets of image data.

[0038] In this embodiment of the invention, step S1 specifically includes the following steps:

[0039] Step S101, data preprocessing for the three modalities includes:

[0040] The T1-weighted structural MRI preprocessing involves segmenting the image into white matter, gray matter, and cerebrospinal fluid images. The gray matter image is then input into an individualized structural brain network to obtain 116*116 structural features based on an Automatic Anatomical Labeling (AAL) template containing 116 brain regions. These structural features represent the similarity of brain regions as measured by the Kullback-Leibler divergence (KL) measure.

[0041] Functional MRI preprocessing included: deleting the signals from the first 10 unstable time points, performing time-to-time correction, head motion correction, registration, normalization to the Montreal Neurological Institute (MNI) space, denoising, extracting the average time series of each brain region based on the AAL template, and finally calculating the Pearson correlation coefficient for the average time series of any two brain regions to obtain a 116*116 functional connectivity matrix as a feature.

[0042] DTI preprocessing includes: removing skull and non-brain tissues, correcting head motion and vortex loss distortion, calculating and fitting the diffusion tensor measure in each voxel, registering, and normalizing to MNI space to obtain global deterministic fiber bundles of the brain. Based on the AAL template, the average score anisotropy of the links between nodes is defined as the connection weight in the DTI network to obtain a 116*116 DTI connection network as a feature.

[0043] In step S102, 116 nodes correspond to 116 brain regions. The feature vectors of the corresponding brain regions obtained from the preprocessing of three sets of MRI images are used as node features, and the correlation measure between the corresponding brain regions is used as the feature of the edges between nodes to construct graph data containing 116 nodes. That is, each individual corresponds to three graph data.

[0044] Step S2: Simultaneously input the three graph data corresponding to the individual into the feature extraction and fusion module of the multimodal brain age prediction model (AgeGNN).

[0045] If it is the training phase, the dataset is divided into training and test sets; otherwise, no division is needed. The three graph data points for each subject are used as inputs to three parallel GNN branches, corresponding one-to-one.

[0046] Set the hyperparameters for feature extraction and fusion GNN based on self-attention, with a head count of 1. In each GNN branch, the features of the brain network nodes are aggregated and updated sequentially through an encoder and decoder composed of three sets of TransformerConv, BatchNorm, and ReLU three-layer architectures, respectively, while ensuring distribution stability. The latent space representation between the encoder and decoder is then used as the feature extracted by that branch.

[0047] Next, the features of the three branches corresponding to the three modalities obtained in the above steps are concatenated through a Concat layer to obtain the fused features (i.e., multi-modal fusion feature vector), thus realizing feature fusion.

[0048] Step S3: The multi-mode fusion feature vector output by the feature extraction and fusion module is input into the prediction module of the AgeGNN model (using MLP) to obtain brain age prediction results.

[0049] In this embodiment of the invention, the fused features are nonlinearly transformed using an MLP composed of Linear, ReLU, BatchNorm, and Linear to obtain the predicted brain age;

[0050] To accurately obtain the prediction performance of the AgeGNN model of this invention, this embodiment uses three performance metrics—MAE, root mean square error, and Pearson correlation between predicted age and actual age—to evaluate the prediction performance of the AgeGNN model constructed in this embodiment. For the training phase, the prediction loss L using MSE error is used. pred and reconstruction loss L recon,j The loss function L, composed of L2 regularization terms, is used to train the AgeGNN model to obtain its parameters. The batch size is set to 16, the initial learning rate to 0.001, and the weight decay to 0.001. The Adam optimizer and cosine annealing learning rate adjustment algorithm are used. Otherwise, the steps do not need to be repeated. The specific formula for the loss function is as follows:

[0051]

[0052] Where N is the number of samples, M is the number of modalities, i = 1...N, j = 1...M, x i,j For the input of the autoencoder, BA is the output obtained by reconstructing the input from the autoencoder. i To predict the obtained brain age, CA i λ represents actual age (i.e., chronological age). j λ1 is a hyperparameter, μ is the regularization coefficient, and w is the model parameter. In this embodiment, M = 3, λ1 = λ2 = λ3 = μ = 1.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0054] The above descriptions are merely some embodiments of the present invention. For those skilled in the art, various modifications and improvements can be made without departing from the inventive concept of the present invention, and all such modifications and improvements fall within the scope of protection of the present invention.

Claims

1. A multimodal medical image prediction method based on graph neural networks, characterized in that, Includes the following steps: Step 1: Perform data preprocessing on the MRI images of various modalities acquired by the input magnetic resonance imaging instrument, and then construct graph data of each modality from the preprocessed MRI image data to obtain graph data of each modality. The MRI images include multiple modalities such as T1-weighted structural MRI, functional MRI, and diffusion tensor imaging (DTI). T1-weighted structural MRI preprocessing involves segmenting the image into white matter, gray matter, and cerebrospinal fluid images. The gray matter images are then input into an individualized structural brain network to obtain 116*116 structural features based on an automated anatomical labeling template containing 116 brain regions. These structural features represent the similarity of brain regions as measured by the KL divergence. Functional MRI preprocessing was performed as follows: the first 10 unstable time point signals were deleted, time-layer correction, head motion correction, registration, normalization to the Montreal Neuroscience Institute space, and denoising. The average time series of each brain region was extracted based on the AAL template. Finally, the Pearson correlation coefficient was calculated for the average time series of any two brain regions to obtain a 116*116 functional connectivity matrix as a feature. DTI preprocessing includes: removing skull and non-brain tissues, correcting head motion and eddy current distortion, calculating and fitting diffusion tensor measurements in each voxel, registering and normalizing to MNI space to obtain global deterministic fiber bundles of the brain, and defining the average score anisotropy of inter-node links based on the AAL template as the connection weights in the DTI network to obtain a 116*116 DTI connection network as a feature. Step 2: Construct a multimodal brain age prediction model based on graph neural networks. This prediction model includes a feature extraction and fusion module and a prediction module. The feature extraction and fusion module includes several feature extraction branches, the number of which is consistent with the number of modalities of the MRI images acquired in step 1. Each feature extraction branch includes a set of encoders and decoders. The encoders and decoders have the same network structure, each including several unit modules composed of graph transform convolutions, batch normalization layers, and activation functions. The encoder's input is single-modality graph data, which extracts features from the graph data based on downsampling. The decoder's input is the encoder's output, which reconstructs the single-modality graph data based on upsampling. All feature extraction branches have the same number of unit modules in their autoencoders, and the extracted features have the same dimension. The feature extraction and fusion module fuses the feature vectors output by the encoders of each feature extraction branch through the splicing layer to obtain a multi-mode fused feature vector, which is then input into the prediction module. The prediction module uses a multilayer perceptron to perform a nonlinear transformation on the multimodal fusion feature vector to obtain the predicted brain age; Step 3: Optimize the model parameters of the multimodal brain age prediction model based on the training data. When the preset training termination condition is met, obtain the age predictor for multimodal medical images based on the autoencoder of each feature extraction branch that has been trained, as well as the splicing layer and prediction module. Among them, the loss function during optimization training includes the reconstruction loss between the encoder input and the graph data of the single modality reconstructed by the decoder for each feature extraction branch, and the prediction loss between the predicted brain age and the label output by the prediction module. The loss function for optimizing training is set as follows: ； in, This represents the loss function used during optimization training. Indicates the predicted loss. Let j represent the reconstruction loss, j represent the modality identifier, N represent the number of samples, and M represent the number of modalities. The coefficient of the regularization term, Here are the model parameters for a multimodal brain age prediction model, where i represents the sample index. This indicates that the decoder reconstructs the single-modal graph data. This represents the input of the autoencoder. Indicates a prediction of brain age. Indicates the actual age.

2. The method as described in claim 1, characterized in that, The prediction module consists of a linear layer, a ReLU activation function, a batch normalization layer, and another linear layer.

3. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 2.

4. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 2.

5. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 2.

Images