A parkinson disease detection model and system based on multi-modal image fusion

Abstract

The application discloses a Parkinson disease detection model and system based on multi-modal image fusion, which comprises an image feature extraction module, a multi-modal fusion module, a prediction module and an output module; after obtaining sMRI, DTI and fMRI images of a user to be detected, different methods are used to extract features and fuse the data of the three channels to obtain multi-modal fusion features of double modes and three modes, and then an integrated learning method is used to finally obtain a diagnosis result for reference by doctors in diagnosis assistance. For the current early diagnosis of Parkinson, the application quantitatively analyzes multi-modal data, has important significance for early diagnosis of Parkinson disease, can improve the accuracy of diagnosis, reduce the subjective judgment error of operators, and has application value for clinical early intervention and reduction of later disability.

Description

Technical Field

[0001] This invention relates to the field of information technology, and more specifically, to a Parkinson's disease detection model and system based on multimodal image fusion. Background Technology

[0002] Parkinson's disease (PD) affects millions of people worldwide and is characterized by motor impairments, restricted behavioral patterns, and communication difficulties. Patients exhibit heterogeneity in behavioral, genetic, and neuroanatomical aspects. More specific diagnosis of PD and targeted interventions to understand its neuroanatomical heterogeneity are key to improving patients' quality of life.

[0003] Magnetic resonance imaging (MRI) has greatly enhanced our understanding of the underlying neural basis of cognitive abilities and plays a crucial role in the clinical diagnosis and efficacy evaluation of patients with Parkinson's disease (PD). Traditional neuroscience research primarily focuses on establishing mapping relationships between imaging metrics and behavioral variables, typically employing univariate analysis to explore the correlation between brain imaging features (such as gray matter volume, cortical thickness, and functional connectivity strength) and specific cognitive scores. While these studies can help understand the surface mechanisms of relevant cognitive behaviors and have statistical descriptive significance, they struggle to predict cognitive abilities at the individual level.

[0004] In image feature extraction, most existing technologies use a single method to extract features from different modalities. However, due to the differences in dimensionality between different modalities, the challenge of multimodal image recognition lies in how to extract and integrate relevant information from different modalities. Existing technologies simply connect the multimodal features in the original space, ignoring the underlying relationships between them. Therefore, they cannot effectively extract all discriminative features, thus limiting the performance of the final PD detection system and hindering its ability to assist doctors in clinical decision-making. Summary of the Invention

[0005] To address the lack of accurate PD detection systems in existing technologies, this invention provides a Parkinson's disease detection model and system based on multimodal image fusion.

[0006] The first objective of this invention is to provide a Parkinson's disease detection model device based on multimodal image fusion.

[0007] A second objective of this invention is to provide the application of the aforementioned Parkinson's disease detection model device in the construction of a Parkinson's disease detection system.

[0008] A third objective of this invention is to provide a Parkinson's disease detection system.

[0009] The fourth object of the present invention is to provide a computer device.

[0010] A fifth object of the present invention is to provide a computer-readable storage medium.

[0011] To achieve the above objectives, the present invention is implemented through the following solution: Firstly, this invention is based on the premise that PD-specific features are related to clinical symptoms, while shared features are related to non-clinical symptoms. It aims to mine neural markers from multimodal images of PD patients to achieve high-precision prediction of motor ability and speech integration ability of PD patients. Specifically, it is as follows: (1) Automatic extraction of multimodal image features: For image data of different dimensions, low-dimensional data is extracted using Inception. The V4 method extracts image features, and high-dimensional data uses pulsed convolutional networks (SNN) to extract image features; (2) The FusNet module intelligently fuses functional and structural image features, which includes the joint architecture fusion model FiANet and the collaborative architecture fusion model MMTM. MMTM is used to calculate mutual attention between two different images (called cross-attention), thereby achieving fusion between all two types of images; In order to learn the high-order feature representations of three different images, this invention uses the FiANet method to form a triplet of three MRIs, including functional-structural fusion features and structural-structural fusion features, to improve the accuracy of the detection system; (3) Adaptive training is performed on the PPMI public dataset to obtain a motor ability prediction model, and its parameters are transferred to the SYU dataset for predicting the motor ability and speech integration ability of PD patients; (4) The category activation map (Grad-CAM) technique is used for feature visualization.

[0012] To better characterize PD-specific neuroanatomical variations, this invention proposes a multimodal transfer learning (TL-MF) model to separate these variations from those common to the subject population. TL-MF takes two distinct behavioral symptoms (motor symptoms and speech disorders) and multimodal neuroimaging features as inputs to isolate population-specific variations from common variations. TL-MF is used to isolate PD-specific neuroanatomical features from the "shared" variations of motor symptoms and speech disorders. This invention uses transfer learning to obtain a generalized deep learning model on the Independent Public Dataset (PPMI) and performs feature fusion on multimodal neuroimaging data to learn functional and structural neuroanatomical information. This invention leverages the characteristics of TL-MF to predict brain regions showing systemic changes in motor symptoms and speech disorders in the PD population, contributing to personalized diagnosis, prognosis, and treatment selection.

[0013] A Parkinson's disease detection model device based on multimodal image fusion includes an image feature extraction module, a multimodal fusion module, a prediction module, and an output module; The image feature extraction module is used to collect data from the structural magnetic resonance imaging, diffusion tensor imaging and functional magnetic resonance imaging images of the brain of the user to be tested, and extract three single-modal features from them in sequence, and then present them to the multimodal fusion module. The three monomodal features are structural magnetic resonance imaging (sMRI) monomodal features, diffusion tensor imaging (DTI) monomodal features, and functional magnetic resonance imaging (fMRI) monomodal features; the brain includes the cerebrum and cerebellum; the data acquired from the images of structural magnetic resonance imaging and diffusion tensor imaging are structural data, and the data acquired from the images of functional magnetic resonance imaging are functional data; The multimodal fusion module is used to fuse the three single-modal features to obtain multimodal fusion features. These multimodal fusion features include bimodal fusion features and trimodal fusion features. The bimodal fusion features are obtained by fusing the three single-modal features pairwise, namely, structural magnetic resonance imaging-diffusion tensor imaging bimodal fusion features (sMRI-DTI bimodal fusion features, 2fusTD), structural magnetic resonance imaging-functional magnetic resonance imaging bimodal fusion features (sMRI-fMRI bimodal fusion features, 2fusTB), and diffusion tensor imaging-fMRI bimodal fusion features (DTI-fMRI bimodal fusion features, 2fusDB). The trimodal fusion features are obtained by fusing the three single-modal features together, namely, structural magnetic resonance imaging-diffusion tensor imaging-fMRI trimodal fusion features (sMRI-DTI-fMRI trimodal fusion features, 3fus). The dual-modal fusion feature is obtained by fusion using a collaborative architecture, and the trimodal fusion feature is obtained by fusion using a joint architecture. The joint architecture includes a soft attention mechanism and a fused attention mechanism; The formula for the soft attention mechanism is as follows: ; ; ; in, u This represents the first fusion feature in the shared subspace; α and β The value ranges from 0 to 1; σ(x) This represents the Sigmoid function; [.,.] represents the concatenation operation; m 1～3 The single-modal characteristics of structural magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging are represented in sequence. , and The features obtained by convolution operation represent the single-modal features of the structured magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging, respectively. C , and These represent the number of channels in the images obtained from the structural magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging, respectively. N i Indicates spatial dimension; The formula for the fusion attention mechanism is as follows: ; ; ; in, Representing modes and Conduct joint learning, This represents the cross product operation; The joint architecture fusion also includes the output of fusion features, as shown in the following formula: ; in, C Indicates the convolution operation; The prediction module is used to obtain prediction results by utilizing the speech ability prediction model and the multimodal fusion features. The prediction results include the behavioral score of the Parkinson's disease of the user to be tested. The speech ability prediction model is obtained by transfer learning using the motor ability prediction model as the source domain. The motor ability prediction model is trained using a Parkinson's disease dataset as the training set in a deep learning framework. The Parkinson's disease dataset includes structural magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging images of Parkinson's patients, as well as behavioral scores for Parkinson's disease. The output module is used to output the prediction result.

[0014] Preferably, the Parkinson's disease detection model uses PyTorch as a deep learning framework.

[0015] Preferably, the image feature extraction module uses a neural network to collect the data and extracts single-modal features from it.

[0016] More preferably, the neural network includes an Inception network and a spiking neural network. The Inception network is used to acquire structural data from structural magnetic resonance imaging (MRI) images of the user under test, and the spiking neural network is used to acquire structural data from diffusion tensor imaging (DTI) images of the user under test and functional data from functional magnetic resonance imaging (fMRI) images.

[0017] More preferably, the Inception network is Inception V4, and the specific formula is as follows: ; in, Slices of structural magnetic resonance imaging images l Changhe w Wide image. C i Indicates size as i Convolution operation, Res This indicates residual network operations.

[0018] More preferably, the formula for the spiking neural network is: ; ; in, This refers to images from diffusion tensor imaging or functional magnetic resonance imaging (fMRI). t Time slices l long w Width h high; Indicates the time interval as The spiking neuron operation; B represents the value of the time dimension, ranging from 1 to 240; D represents the value of the diffusion direction dimension, ranging from 1 to 65.

[0019] More preferably, the collaborative architecture fusion includes compressing and activating the three unimodal features.

[0020] More preferably, the specific formula for the fusion of the collaborative architecture is as follows: ; ; ; in, , and The features obtained by convolution operation represent the single-modal features of the structured magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging, respectively. C , and These represent the number of channels in the images obtained from the structural magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging, respectively. N i Indicates spatial dimension.

[0021] More preferably, the collaborative architecture fusion further includes assigning weights to each single modality, using the following formula: ; ; in, , and The activation function represents the multimodal image. Calculate the common compressed subspace of multimodal images; [.,.] denotes the concatenation operation; , , and Represents a constant parameter matrix; , , and This indicates the deviation of the fully connected layer in the collaborative architecture integration.

[0022] Preferably, the Parkinson's disease dataset includes the Parkinson's disease dataset in the prior art “DOI: 10.1016 / j.pneurobio.2011.09.005”.

[0023] Preferably, the specific formula for training the athletic ability prediction model is as follows: ; in, This represents the motor ability score of the i-th subject in the training set. u i Representing multimodal fusion features, through Calculation parameters e i , express x i Probability distribution; S represents the source domain, and T represents the target domain; The The calculation formula is: , where N represents the number of samples.

[0024] More preferably, the transfer learning method includes optimizing the balance between the source and target domains in the Hilhert space using a kernel mean matching algorithm, as shown in the following formula: XVIII: ; in, e i From the formula Calculated B and d is a constraint constant.

[0025] Preferably, the behavioral score for Parkinson's disease includes a motor ability score and / or a language ability score.

[0026] More preferably, the behavioral scores for Parkinson's disease include UPDRS2, UPDRS3, and / or VoReMa.

[0027] Preferably, the prediction results also include biomarkers in the brain of the test user that are related to behavioral scores of Parkinson's disease.

[0028] More preferably, the output module also visualizes and outputs the biomarkers.

[0029] More preferably, the visualization method includes the following steps: building a network using torch, setting "requires_grad=True" in the ResNet toolkit, and inputting the prediction results to obtain the result.

[0030] Preferably, the Parkinson's disease detection model further includes a storage module for storing the structural data, the functional data, the three unimodal features, the multimodal fusion features, the motor ability prediction model, the speech ability prediction model, and / or the prediction results.

[0031] More preferably, the storage module also stores training parameters for the motor ability prediction model and training parameters for the speech ability prediction model.

[0032] The application of any of the Parkinson's disease detection model devices in the construction of Parkinson's disease detection systems should also be within the scope of protection of this invention.

[0033] A Parkinson's disease detection system includes an acquisition unit, a storage unit, and a processing unit. The acquisition unit is used to acquire structural magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging images of the brain of the user to be tested. The storage unit stores program instructions that can be executed by the processing unit. The program instructions input the structural magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging images of the brain of the user to be tested, which are obtained by the acquisition unit, into the processing unit. The processing unit includes any of the Parkinson's disease detection model devices described above, and executes the program instructions to obtain Parkinson's disease detection results.

[0034] Preferably, the storage unit further stores the structural data, functional data, the three unimodal features, the multimodal fusion features, the motor ability prediction model, the speech ability prediction model, and / or the prediction results obtained by the processing unit.

[0035] More preferably, the storage unit also stores training parameters of the motor ability prediction model and / or training parameters of the speech ability prediction model.

[0036] A computer device includes a memory and a processor, wherein the memory stores a computer program executable on the processor; when the computer program is executed by the processor, it implements the operation of the Parkinson's disease detection system.

[0037] A computer-readable storage medium storing a computer program executable by a processor, wherein the computer program, when executed by the processor, implements the operation of the Parkinson's disease detection system.

[0038] A computer program product, wherein the computer program product is a computer-readable storage medium.

[0039] Compared with the prior art, the present invention has the following beneficial effects: This invention provides a Parkinson's disease detection model and system based on multimodal image fusion. After obtaining sMRI, DTI, and fMRI images of the user, different methods are used to extract and fuse multimodal fusion features. Then, an ensemble learning method is used to obtain the final diagnostic result, which can be used by doctors for diagnostic assistance. Regarding the early diagnosis of Parkinson's disease, this invention quantitatively analyzes multimodal data, which is of great significance for the early diagnosis of Parkinson's disease. It can not only improve the accuracy of diagnosis and reduce the subjective judgment error of the operator, but also has application value for early clinical intervention and thus early treatment. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating a Parkinson's disease detection model based on multimodal image fusion according to the present invention.

[0041] Figure 2 This is a saliency diagram of the TL-MF model of this invention. Detailed Implementation

[0042] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. These embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Unless otherwise specified, the experimental methods used in the following embodiments are conventional methods; the materials and reagents used, unless otherwise specified, are commercially available.

[0043] Example 1: A Parkinson's Disease Detection Model Based on Multimodal Image Fusion The Parkinson's disease detection model provided by this invention includes an image feature extraction module, a multimodal fusion module, a prediction module, an output module, and a storage module, and its operation flow is as follows: Figure 1As shown.

[0044] In summary, the Parkinson's disease detection model of this invention uses PyTorch as a deep learning framework. First, it automatically collects structural and functional data from the brain MRI images of the test user using Inception V4 and spiking neural networks (SNN) deep learning techniques, and extracts structural and functional features related to Parkinson's disease in sequence. Then, it uses the FusNet module to fuse features between different modalities, which includes FiANet and MMTM methods. Next, it adaptively trains on existing publicly available datasets to establish a motor ability prediction model, and then transfers the parameters of this model to the test user's dataset to predict the test user's speech integration ability. Finally, it uses Class Activation Map (CAM) technology to visualize the prediction results.

[0045] The specific method for constructing the Parkinson's disease detection model is as follows: 1. Image Feature Extraction Module The image feature extraction module of this model is used to collect data from the structural magnetic resonance imaging, diffusion tensor imaging and functional magnetic resonance imaging images of the brain (cerebrum and cerebellum) of the test user, and extract features from them in sequence to obtain three single-modal features, which are then presented to the multimodal fusion module for modal fusion.

[0046] (1) Inception V4 processing of structural magnetic resonance imaging (sMRI) sMRI is excellent at measuring the fluid characteristics of different tissues (white matter and gray matter), with a resolution down to the millimeter level, strong gray / white matter contrast, and images with many anatomical details. While classic voxel-based morphology analysis (VBM) methods can obtain morphological features of brain regions on sMRI, they are often subject to the segmentation prior of brain region template selection, ignoring inter-individual differences.

[0047] The core of Inception V4 is its ability to adaptively detect image features at different scales, aiming to solve the gradient explosion or vanishing gradient problems encountered by VGG networks in deep learning. This technique uses different receptive fields to acquire image features in a parallel manner, offering advantages such as simple design and fast execution speed, and is applicable to both 2D and 3D images.

[0048] This invention utilizes Inception V4 to collect structural data (low-dimensional data) of sMRI image slices of the brain (cerebrum and cerebellum) of the test user, and extracts structural features related to Parkinson's disease. The specific principle is shown in formula (1): Formula (1): ; in, Slices of structural sMRI images l Changhe w Wide image. C i Indicates size as i Convolution operation, Res This indicates residual network operations.

[0049] (2) Pulsed convolutional networks (SNNs) are used to process diffusion tensor imaging (DTI) or functional magnetic resonance imaging (fMRI). Originating in computer neuroscience, Split Neural Networks (SNNs) have garnered widespread attention in brain-inspired computing and neuromorphic engineering due to their diverse encoding mechanisms and rich spatiotemporal dynamics. Currently, mainstream SNN algorithms include unsupervised training algorithms represented by temporally dependent plasticity (STDP) and supervised training algorithms represented by backpropagation of errors in the temporal and spatial dimensions. SNNs, based on the event-driven mechanism of neurodynamics, have achieved outstanding performance in handling sparse, complex, and noisy spatiotemporal information.

[0050] Compared to sMRI images, DTI and fMRI have higher dimensions and possess temporal or diffusion direction characteristics. This invention uses a combination of deep learning and STDP algorithms to collect structural data from DTI images and functional data from fMRI images of the brain (cerebrum and cerebellum) of the test subject (both are high-dimensional data), and extracts functional features related to Parkinson's disease. The specific principles are shown in formulas (2) and (3): Formula (2): ; Formula (3): ; in, Images representing an individual's DTI or fMRI, the images t Time slices l long w Width h high; Indicates the time interval as The spiking neuron operation; B represents the value of the time dimension, ranging from 1 to 240; D represents the value of the diffusion direction dimension, ranging from 1 to 65.

[0051] Based on the above method, the image feature extraction module extracts three corresponding monomodal features (i.e., sMRI monomodal feature (sinT1), DTI monomodal feature (sinDTI), and fMRI monomodal feature (sinBold)) from the sMRI, DTI, and fMRI images of the user under test. The obtained monomodal features are stored in the storage module and are used to present them to the multimodal fusion module for further fusion processing.

[0052] 2. Multimodal Fusion Module (MF Module) The challenge of multimodal image recognition lies in how to effectively extract and integrate relevant complementary information from different modalities. However, most existing methods simply connect the multimodal features in the original space (joint methods and alignment methods) while ignoring the underlying relationships between them.

[0053] Considering the global heterogeneity gap between modalities and the local consistency of each modality, and in order to obtain higher-level feature representations of the three single modalities, this invention introduces a joint architecture fusion model (FiANet) and a collaborative architecture fusion model (MMTM) to construct the FusNet module, which integrates the advantages of FiANet and MMTM. The self-attention fusion features learned by the FusNet module are fed into a fully connected network, which generates an optimized representation vector through loss function optimization. The FuseNet module utilizes the self-attention mechanism to extract salient features from the three MRI inputs, while the MMTM of FusNet is used to compute cross-modal attention to achieve fusion of all modalities.

[0054] The multimodal fusion module of this model is used to fuse the three single-modal features obtained by the image feature extraction module.

[0055] (1) MMTM method - dual-modal fusion Since the feature vectors of different modalities are initially located in different subspaces, there is a heterogeneity gap, which hinders the further improvement of the model and the comprehensive utilization of multimodal data in subsequent deep learning models. This problem can be solved by projecting heterogeneous features into a common subspace.

[0056] The MMTM method aims to learn a common subspace by directly measuring the distance between vectors and different modalities, thereby maximizing the relevance of representation sets across different modalities. Under the constraint of similarity measurement, it preserves the similarity structure between and within modalities, aiming to minimize the cross-modal similarity distance for objects with the same semantics or related relationships, while maximizing the distance between objects with different semantics.

[0057] This invention employs the MMTM method to perform pairwise compression and activation operations on the sMRI single-modal features, DTI single-modal features, and fMRI single-modal features obtained from the image feature extraction module, and then fuses them to obtain binary multimodal features.

[0058] Compression is achieved through average pooling to fuse images of different modalities. The specific principle is shown in formulas (4) to (6): Formula (4): ; Formula (5): ; Formula (6): ; in, , and The convolution operations are represented sequentially for the sMRI single-modal features, DTI single-modal features, and fMRI single-modal features obtained from the image feature extraction module; C , and These represent the number of channels in the sMRI, DTI, and fMRI images obtained from the image feature extraction module, respectively. N i Indicates spatial dimension.

[0059] To further improve prediction accuracy, this module also features a weight reallocation function, used to redistribute the weights of the sMRI single-modality, DTI single-modality, and fMRI single-modality, i.e., the weights of the structural modality (A: sMRI; B: DTI) and the functional modality (C: fMRI). E A , E B and E C The specific principles are shown in formulas (7) and (8): Formula (7): ; Formula (8): ; in, , and The activation function represents the multimodal image. Calculate the common compressed subspace of multimodal images; [.,.] denotes the concatenation operation; , , and Represents a constant parameter matrix; , , and This indicates the deviation of the fully connected layer in MMTM.

[0060] Based on the MMTM method described above, three sets of bimodal fusion features were finally obtained: sMRI-DTI bimodal fusion features (2fusTD), sMRI-fMRI bimodal fusion features (2fusTB), and DTI-fMRI bimodal fusion features (2fusDB).

[0061] The obtained three sets of dual-modal fusion features are stored in the storage module for subsequent presentation to the prediction module for further processing.

[0062] (2) FiANet method - three-modal fusion To optimize the performance of FusNet through multimodal image feature fusion, this invention also introduces the FiANet method, which projects the sMRI single-modal features, DTI single-modal features, and fMRI single-modal features obtained from the image feature extraction module into a shared semantic subspace, which is then fused to obtain a set of ternary multimodal features.

[0063] The FiANet method includes soft attention mechanisms and fused attention mechanisms.

[0064] The soft attention mechanism assigns different weights to images of different modalities and superimposes them. Its specific principle is shown in formulas (9) to (11): Formula (9): ; Formula (10): ; Formula (11): ; in, u This represents the first fusion feature in the shared subspace; α and β The value ranges from 0 to 1; σ(x) This represents the Sigmoid function; [.,.] represents the concatenation operation; m i Representing different modal images, m 1～3 These represent sMRI, DTI, and fMRI, respectively.

[0065] The fusion attention mechanism is a cross-fusion of information from multiple modalities. Its specific principle is shown in formulas (12) to (14): Formula (12): ; Formula (13): ; Formula (14): ; in, Representing modes and Conduct joint learning, This indicates the cross product operation.

[0066] The fusion feature output method of FiANet is shown in Equation XV: XV: ; in, C This indicates a convolution operation.

[0067] Based on the FiANet method described above, the single-modal features of the three MRI (sMRI, DTI, and fMRI) are finally combined into a triplet of multimodal fusion features, namely sMRI-DTI-fMRI trimodal fusion features (3fus), to improve the accuracy of the detection system.

[0068] The obtained trimodal fusion features are stored in the storage module for subsequent presentation to the prediction module for further processing.

[0069] 3. Prediction module (i.e., TL module) Traditional deep learning methods require that the training and test sets be independent and identically distributed. When generalizing out of distribution, a significant domain shift occurs between the training and test sets, thus limiting the further development of deep learning. Transfer learning, on the other hand, can uncover common features in two different but related domains. It is a form of imitation learning, enabling users to "apply knowledge to new situations" and improve problem-solving efficiency by drawing on existing knowledge or experience.

[0070] Transfer learning, based on the domain adaptation assumption of data distribution, assumes that the feature space and category space of the data are consistent, while the condition and marginal distributions between the source and target domains differ. It addresses the task variation problem by reducing the distribution differences between domains.

[0071] This module establishes a motor ability prediction model based on existing datasets through pre-training, and then uses transfer learning technology to transfer the motor ability prediction model to obtain a speech ability prediction model. Finally, it uses the speech ability prediction model and the bimodal and trimodal features obtained by the multimodal fusion module to obtain the corresponding prediction results.

[0072] This module employs a learning architecture of the same depth for both the source domain (i.e., the motor ability prediction model) and the target domain (i.e., the speech ability prediction model), thereby achieving both global domain alignment and consideration of intra-class similarity, without causing the loss of fine-grained information for each category.

[0073] (1) Establishment of motor ability prediction model and verbal ability prediction model This invention uses the publicly available Parkinson's disease dataset (i.e., PPMI, which includes sMRI, DTI, fMRI, and behavioral scores (UPDRS2, UPDRS3, and VoReMa)) as the training set, and pre-trains it in a deep learning framework (PyTorch) to establish a motor ability prediction model. The specific principle is shown in formula (15): Formula (15): ; in, This represents the motor ability score of the i-th subject in the publicly available PPMI dataset. Representing multimodal fusion features, through Calculation parameters e i , express x i Probability distribution. S represents the source domain, and T represents the target domain.

[0074] The result is obtained by formula (16): Formula (16): ; Where N represents the number of samples.

[0075] In the Hilhert space, the kernel mean matching algorithm is used to optimize the balance between the source and target domains, and a speech ability prediction model is established. The specific principle is shown in formula (17): Formula (17): ; in, e i Calculated using formula (16), B and d is a constraint constant.

[0076] The obtained motor ability prediction model and its training parameters, and the speech ability prediction model and its training parameters are stored in the storage module.

[0077] (3) Prediction By retrieving multimodal fusion features and inputting them into the motor ability prediction model and the speech ability prediction model, the prediction results for the user under test are obtained, including: motor ability score, speech ability score, and biomarkers related to brain behavioral scores for Parkinson's disease (UPDRS2, UPDRS3, and VoReMa).

[0078] The predicted results are stored in the storage module.

[0079] 4. Output module This module is used to output the prediction results.

[0080] To facilitate clinical analysis and diagnosis, this module also visualizes the biomarkers related to the behavioral scores (UPDRS2, UPDRS3, and VoReMa) of the test user's brain in the prediction results by setting the parameters of the inverse gradient in PyTorch. The specific method is as follows: build a network using torch, set "requires_grad=True" in the ResNet toolkit, and input the prediction results.

[0081] 5. Storage module The storage module of this invention is used to store the following information: structural data, functional data, sMRI monomodal features, DTI monomodal features, and fMRI monomodal features in the image feature extraction module; multimodal fusion features (sMRI-DTI bimodal fusion features, sMRI-fMRI bimodal fusion features, DTI-fMRI bimodal fusion features, and sMRI-DTI-fMRI trimodal fusion features) in the multimodal fusion module; and motor ability prediction models and their training parameters, speech ability prediction models and their training parameters, and prediction results (motor ability scores, speech ability scores, and biomarkers related to brain behavioral scores for Parkinson's disease (UPDRS2, UPDRS3, and VoReMa)) in the prediction module.

[0082] Example 2: Performance Comparison of Deep Learning Models 1. Experimental Methods Taking the Parkinson's disease dataset (PPMI, from: DOI: 10.1016 / j.pneurobio.2011.09.005) as an example, the performance of the deep learning model provided by this invention—TL-MF model (i.e., TL module + MF module)—is compared with that of existing deep learning models by the similarity between the predicted and actual behavior scores. The specific evaluation metrics are: mean absolute error (MAE) of UPDRS3, CS2 (probability of MAE < 2) and CS3 (probability of MAE < 3).

[0083] 2. Experimental Results As shown in Table 1, it is clear that the SNN and Inception-v4 deep learning models outperform other models, but the TL-MF model performs best, with the smallest MAE and the largest CS2 and CS3.

[0084] Table 1. Performance of deep learning models predicting UPDRS3 using the PPMI dataset (T1 images).

[0085] Example 3: Ablation Experiment of TL-MF Model 1. Experimental Methods Using the Parkinson's disease dataset (PPMI, from: DOI: 10.1016 / j.pneurobio.2011.09.005) as the training set and the SYU dataset (clinical data of Parkinson's patients collected from the First Affiliated Hospital of Sun Yat-sen University) as the test set, the Parkinson's disease detection model provided in Example 1 of this invention was used for prediction to evaluate the effectiveness of the MF module and TF module in transfer learning and feature fusion. The specific evaluation metrics were: mean absolute error (MAE) of UPDRS2, UPDRS3 and VoReMa, CS2 (probability of MAE < 2) and CS3 (probability of MAE < 3).

[0086] 2. Experimental Results Table 2 shows the performance of TL-MF in transfer learning and feature fusion.

[0087] (1) Validity of the MF module As shown in Table 2, the three-modal fusion technique exhibits superior performance, with CS2 and CS3 having lower MAE values, closer to 1; the introduction of the MF module reduced the MAE of the deep learning model by 2.047. Furthermore, in terms of single modalities, the functional modality (sinBold) outperforms the structural modality (sinDTI and sinT1).

[0088] (2) Validity of TL module As shown in Table 2, when the MAE values ​​of motor and verbal scores are low, the trimodal 3fus performs better after introducing the TL module, with the MAE decreasing by 0.568 compared to the version without the TL module.

[0089] Saliency map of the TL-MF model in Example 4 1. Experimental Methods Using the Parkinson's disease dataset (PPMI, from: DOI: 10.1016 / j.pneurobio.2011.09.005) as the training set and the SYU dataset (clinical data of Parkinson's patients collected from the First Affiliated Hospital of Sun Yat-sen University) as the test set, the Parkinson's disease detection model provided in Example 1 of this invention was used to predict the image biomarkers of motor symptoms (UPDRS2 and UPDRS3) and speech symptoms (VoReMa, speech integration ability), and to generate visualized images.

[0090] 2. Experimental Results like Figure 2As shown, the red area represents the distribution of three-modal fusion features (i.e., 3fusM), and the darker the color, the greater the contribution of this area to the TL-MF model; the blue area represents the distribution of two-modal fusion features (i.e., 2fusM) (selecting the better modality fusion), and the darker the color, the greater the contribution of this area to the TL-MF model; the green area represents the distribution of unimodal features (i.e., sinM) (optimal mode selection), and the darker the color, the greater the contribution of the unimodal features in this area to the TL-MF model; the yellow shaded area shows the distribution of overlapping features in the three cases (i.e., overlap), and the darker the color, the higher the value of the overlapping features in this area in building the TL-MF model.

[0091] Overall, feature extraction revealed that the superior temporal gyrus (STG) and premotor cortex (PMC) in the brain are the most critical and effective regions involved in the progression of Parkinson's disease, which can help clinicians in the diagnosis and analysis of the disease.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description and ideas, and it is neither necessary nor possible to exhaustively describe all implementation methods here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A Parkinson's disease detection model device based on multimodal image fusion, characterized in that, It includes an image feature extraction module, a multimodal fusion module, a prediction module, and an output module; The image feature extraction module is used to collect data from the structural magnetic resonance imaging, diffusion tensor imaging and functional magnetic resonance imaging images of the brain of the user to be tested, and extract features from them in sequence to obtain three single-modal features, which are then presented to the multimodal fusion module. The three single-modal features are structural magnetic resonance imaging (SMRI) single-modal features, diffusion tensor imaging (DTI) single-modal features, and functional magnetic resonance imaging (fMRI) single-modal features; the brain includes the cerebrum and cerebellum; the data acquired from the images of the structural MRI and DTI are structural data, and the data acquired from the images of the fMRI are functional data. The multimodal fusion module is used to fuse the three single-modal features to obtain multimodal fusion features. The multimodal fusion features are bimodal fusion features and trimodal fusion features. The bimodal fusion features are obtained by fusing the three single-modal features in pairs to obtain three bimodal fusion features. The trimodal fusion features are obtained by fusing the three single-modal features together. The dual-modal fusion feature is obtained by fusion using a collaborative architecture, and the trimodal fusion feature is obtained by fusion using a joint architecture. The joint architecture includes a soft attention mechanism and a fused attention mechanism; The formula for the soft attention mechanism is as follows: ； ； ； in, u i Let i represent the i-th fusion feature in the shared subspace, where i = 1, 2, 3, 4; α and β The value ranges from 0 to 1; σ(x) This represents the Sigmoid function; [.,.] represents the concatenation operation; , and The features obtained by convolution operation represent the single-modal features of the structured magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging, respectively. C , and These represent the number of channels in the images obtained from the structural magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging, respectively. N i Indicates spatial dimension; The formula for the fusion attention mechanism is as follows: ； ； ； The joint architecture fusion also includes the output of fusion features, as shown in the following formula: ； in, C ( ) represents the convolution operation; The prediction module is used to obtain prediction results by utilizing the speech ability prediction model and the multimodal fusion features. The prediction results include the behavioral score of the Parkinson's disease of the user to be tested. The speech ability prediction model is obtained by using the motor ability prediction model as the source domain and through transfer learning. The motor ability prediction model is trained using a Parkinson's disease dataset as the training set in a deep learning framework. The Parkinson's disease dataset includes structural magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging images of Parkinson's patients, as well as behavioral scores for Parkinson's disease. The output module is used to output the prediction result.

2. The Parkinson's disease detection model device according to claim 1, characterized in that, The image feature extraction module uses a neural network to collect data from structural magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging images of the brain of the user under test, and extracts features from them sequentially to obtain three types of single-modal features.

3. The Parkinson's disease detection model device according to claim 2, characterized in that, The neural network includes the Inception network and the spiking neural network.

4. The application of the Parkinson's disease detection model device according to any one of claims 1 to 3 in the construction of a Parkinson's disease detection system.

5. A Parkinson's disease detection system, comprising an acquisition unit, a storage unit, and a processing unit, characterized in that, The acquisition unit is used to acquire structural magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging images of the brain of the user to be tested. The storage unit stores program instructions that can be executed by the processing unit. The program instructions input the structural magnetic resonance imaging, diffusion tensor imaging, and functional magnetic resonance imaging images of the brain of the user to be tested, which are obtained by the acquisition unit, into the processing unit. The processing unit includes the Parkinson's disease detection model device according to any one of claims 1 to 3, and executes the program instructions to obtain the Parkinson's disease detection result.

6. The Parkinson's disease detection system according to claim 5, characterized in that, The storage unit also stores the structural data, functional data, three unimodal features, multimodal fusion features, motor ability prediction model, speech ability prediction model, and prediction results obtained by the processing unit.

7. A computer device, comprising a memory and a processor, characterized in that, The memory stores computer programs that can be executed on the processor; When the computer program is executed by the processor, it implements the operation of the Parkinson's disease detection system as described in claim 5 or 6.

8. A computer-readable storage medium storing a computer program executable by a processor, wherein when executed by the processor, the computer program performs the operation of the Parkinson's disease detection system of claim 5 or 6.

Images