Parkinson's disease treatment effect prediction method and device based on multi-modal image model

Abstract

The invention relates to a Parkinson's disease treatment effect prediction method based on a multi-modal image model and a related device. The method comprises the following steps: acquiring a multi-modal vector of a Parkinson's disease patient; aligning the multi-modal vectors on a time axis, and constructing a time point data element sample sequence; deploying a double-flow cross attention encoder for a sample sequence in each time point data element, and outputting a fused multi-modal feature through the double-flow cross attention encoder; and connecting the fused multi-modal features with digital clinical treatment scheme vectors corresponding to corresponding time points to form time point comprehensive feature vectors, inputting the time point comprehensive feature vectors to a time sequence information aggregation gating circulation unit, outputting final time point aggregation features, and inputting the final time point aggregation features to a multi-task adaptive prediction head. A UPDRS total score or a specific sub-scale score for the patient at a future preset point in time is predicted. According to the method, time sequence modeling is carried out on multi-mode and multi-time-point data, so that the accuracy and interpretability of Parkinson's disease treatment effect prediction are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and medical auxiliary technology, and particularly relates to a Parkinson's disease treatment effect prediction method and device based on a multi-modal image model and a computer device. BACKGROUND

[0002] Parkinson's Disease (PD) is a common neurodegenerative disease that mainly affects motor function, with diverse clinical manifestations including resting tremor, bradykinesia, muscle rigidity, and postural instability. Currently, the treatment of PD mainly focuses on symptomatic treatment to improve the quality of life of patients, but there is no cure. Common treatment methods include drug therapy (such as levodopa, dopamine receptor agonists, etc.), deep brain stimulation (DBS), and other non-drug therapies. Traditional PD evaluation methods mainly rely on the experience of clinicians and scale scores, among which the Unified Parkinson's Disease Rating Scale (UPDRS) is the most widely used tool, reflecting disease progression and treatment effect by assessing the severity of motor and non-motor symptoms. However, UPDRS scores have a certain degree of subjectivity, and there may be differences in assessments between different doctors, and they can only reflect the severity of symptoms at a certain time point, making it difficult to capture the dynamic trends during treatment. In addition, the effects of drug therapy vary from person to person, and patients may have significantly different responses to the same drug, and as the disease progresses, some patients may develop motor complications such as dyskinesia and end-of-dose phenomena, further increasing the complexity of treatment effect evaluation.

[0003] PET (positron emission tomography) and MR (magnetic resonance) images play an increasingly important role in the diagnosis and research of Parkinson's disease. PET images can reflect functional information such as neurotransmitter metabolism, receptor density, and inflammatory response in specific brain regions through tracers, providing objective evidence for assessing dopamine neuron dysfunction. MR images provide high-resolution structural information of the brain, including brain atrophy, white matter lesions, and morphological changes in specific brain regions such as the substantia nigra and striatum. For example, high-field MR (such as 7T MR) can more clearly display abnormal signals in the substantia nigra, which is helpful for early diagnosis and differential diagnosis.

[0004] Despite significant advancements in the diagnosis and research of Parkinson's disease (PD), effectively integrating multimodal imaging data with clinical information to achieve dynamic and accurate predictions of treatment outcomes remains a pressing issue. Existing research largely focuses on utilizing single-modal images or simply concatenating features from different modalities, failing to fully explore the intrinsic correlations between multimodal data and their dynamic changes over time. For example, some studies attempt to predict PD progression using baseline imaging features, but few studies have been able to track imaging changes in real time during treatment and integrate them with clinical treatment protocols and efficacy assessment indicators. Furthermore, most deep learning models are black-box models, making it difficult to explain the basis of their predictions to clinicians, thus hindering their application in clinical practice.

[0005] To address the aforementioned issues, this invention proposes a method for predicting the treatment efficacy of Parkinson's disease based on a multimodal image model. By performing time-series modeling on multimodal and multi-time-point data, the accuracy and interpretability of the prediction of Parkinson's disease treatment efficacy are improved. Summary of the Invention

[0006] In view of the above problems, the present invention provides a method and device for predicting the treatment effect of Parkinson's disease based on a multimodal image model, as well as a computer device.

[0007] According to one aspect of the present invention, a method for predicting the treatment effect of Parkinson's disease based on a multimodal imaging model is provided, comprising:

[0008] Obtain multimodal vectors of Parkinson's disease patients at different treatment stages, wherein the multimodal vectors include baseline data, follow-up PET / MR images, and corresponding concurrent clinical treatment plans and multidimensional efficacy evaluation index sequences;

[0009] The multimodal vectors are aligned on the time axis to construct a multimodal and multi-time point time point data element sample sequence; a dual-stream cross-attention encoder is deployed in the sample sequence of each time point data element, and the features of one modality are projected into the feature space of another modality through the dual-stream cross-attention encoder to output fused multimodal features; wherein, each PET stream / MR stream uses an independent 3D convolutional neural network;

[0010] The fused multimodal features are connected with the corresponding digital clinical treatment plan vectors at the corresponding time points to form a time point comprehensive feature vector; the time point comprehensive feature vector is input into a gated loop unit for time-series information aggregation, and the final time point aggregated features are output; the final time point aggregated features are input into a multi-task adaptive prediction head to predict the patient's UPDRS total score or specific subscale score at a future preset time point;

[0011] The UPDRS total score or specific sub-scale score is optimized by minimizing the Wasserstein distance between the predicted trajectory and the actual trajectory, and the contribution of the final time point aggregated features to the final prediction result is quantified by SHAP or LIME method.

[0012] In an optional mode, a wavelet dynamic time warping method based on harmonic analysis is used to align the multi-modal vectors on the time axis, wherein the cost function of the wavelet dynamic time warping method is:

[0013]

[0014] wherein, is the cost between the ith and jth time series; is the s-layer wavelet decomposition operator; is the scale weight of the s-layer; is the phase feature of the treatment response function; is the phase integral constraint coefficient; , is the original data of the ith and jth samples, respectively.

[0015] In an optional mode, the calculation formula of the cross-attention weight matrix of the double-flow cross-attention encoder is:

[0016]

[0017] wherein, is the query matrix of the PET modality; is the key matrix of the MR modality; is the query vector dimension; is the activation function; is the quantum fluctuation adjustment coefficient; is the feature matrix of the PET modality; is the feature matrix of the MR modality; is the super-symmetry exchange matrix; is the matrix trace operator.

[0018] In an optional mode, the gating recurrent unit of the time series information aggregation adopts a fractional calculus enhancement method, and the update rule of the hidden state is:

[0019]

[0020] wherein, is the hidden state at time t; is the input at time t-k; is the calculation result of the gating recurrent unit on the input . is a gamma function; is a fractional derivative order; is a generalized binomial coefficient; is a historical time step.

[0021] In an alternative way, the multi-task adaptive prediction head comprises a plurality of independent prediction branches;

[0022] Each prediction branch is used to predict one of the UPDRS total score or the specific subscale score.

[0023] The multi-task adaptive prediction head dynamically adjusts the weights of each prediction branch according to the importance of different prediction tasks, the distribution characteristics of training data, and the performance of the model in historical predictions.

[0024] In an alternative way, the method further comprises:

[0025] Segmenting and volumetrically measuring the substantia nigra, striatum, and cortex of the PET / MR image to extract morphological, functional, and metabolic quantitative features of the segmented regions.

[0026] In an alternative way, before predicting the UPDRS total score or the specific subscale score of the Parkinson's disease patient at a future preset time point, the method further comprises:

[0027] For PET images, motion correction, attenuation correction, scatter correction, and standardization to a common template space are performed to eliminate individual differences and scan artifacts;

[0028] For MR images, intensity inhomogeneity correction, skull stripping, and registration to the PET image space are performed to ensure accurate alignment of PET and MR images.

[0029] In an alternative way, the method further comprises:

[0030] Collecting tremor frequency and gait stability data of the patient through a mobile application to construct time-series PRO data; wherein the gait stability data includes gait speed, stride length, and gait bilateral symmetry;

[0031] Matching the time-series PRO data with the contemporaneous PET / MR image features on the time axis.

[0032] According to another aspect of the present application, a Parkinson's disease treatment effect prediction device based on a multi-modal image model is provided, comprising:

[0033] A data acquisition module is configured to acquire multi-modal vectors of a Parkinson's disease patient at different treatment stages, wherein the multi-modal vectors include baseline data, follow-up PET / MR images, and corresponding contemporaneous clinical treatment plans and multi-dimensional efficacy evaluation index sequences;

[0034] A multi-modal fusion module is configured to align the multi-modal vectors on a time axis, construct multi-modal and multi-time point time point data element sample sequences, deploy a double-flow cross-attention encoder for each sample sequence in each time point data element, project features of one modality to a feature space of another modality through the double-flow cross-attention encoder, and output fused multi-modal features, wherein each PET flow / MR flow adopts an independent 3D convolutional neural network.

[0035] A time sequence feature aggregation and prediction module is configured to connect the fused multi-modal features and corresponding digitalized clinical treatment plan vectors of the corresponding time points to form time point comprehensive feature vectors, input the time point comprehensive feature vectors into a gated recurrent unit of time sequence information aggregation, output final time point aggregated features, and input the final time point aggregated features into a multi-task adaptive prediction head to predict UPDRS total scores or specific subscale scores of the patient at a future preset time point.

[0036] A prediction optimization and explainability analysis module is configured to optimize the UPDRS total scores or specific subscale scores by minimizing the Wasserstein distance between a predicted trajectory and an actual trajectory, and quantify the contribution of the final time point aggregated features to the final prediction result by SHAP or LIME methods.

[0037] According to another aspect of the present application, a computer device is provided, comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus;

[0038] The memory is used to store at least one executable instruction, and the executable instruction makes the processor execute the operations corresponding to the above-mentioned Parkinson's disease treatment effect prediction method based on a multi-modal image model.

[0039] According to the scheme provided by the application, a multi-modal vector of a Parkinson's disease patient at different treatment stages is obtained, wherein the multi-modal vector includes baseline data, follow-up PET / MR images, and corresponding contemporaneous clinical treatment schemes and multi-dimensional therapeutic effect evaluation index sequences; the multi-modal vector is aligned on a time axis to construct a multi-modal and multi-time point time point data element sample sequence; a double-flow cross-attention encoder is deployed for each sample sequence in each time point data element, the double-flow cross-attention encoder is used to project the features of one modality to the feature space of another modality, and fused multi-modal features are output; wherein each PET flow / MR flow adopts an independent 3D convolutional neural network; the fused multi-modal features are connected with the corresponding digitalized clinical treatment scheme vector at the corresponding time point to form a time point comprehensive feature vector; the time point comprehensive feature vector is input into a gated recurrent unit of time sequence information aggregation, and a final time point aggregated feature is output; the final time point aggregated feature is input into a multi-task adaptive prediction head to predict the UPDRS total score or specific subscale score of the patient at a future preset time point; the UPDRS total score or specific subscale score is optimized by minimizing the Wasserstein distance between the predicted trajectory and the actual trajectory, and the contribution of the final time point aggregated feature to the final prediction result is quantified by the SHAP or LIME method. The application improves the accuracy and interpretability of Parkinson's disease treatment effect prediction by performing time sequence modeling on multi-modal and multi-time point data. Specifically, by aligning the multi-modal data on the time axis to construct a time point data element sample sequence, and using a gated recurrent unit to aggregate time sequence information, the dynamic trajectory of disease development and treatment effect can be captured. By minimizing the Wasserstein distance between the predicted trajectory and the actual trajectory, the actual disease progression path is more accurately fitted, which has important clinical significance for long-term treatment management. By predicting the UPDRS total score or specific subscale score of the patient at a future preset time point, the doctor is provided with a prospective treatment effect evaluation, which helps the doctor to adjust the treatment scheme in a timely manner according to the specific condition of the patient and the expected therapeutic effect.

[0040] The above description is only a summary of the technical scheme of the application. In order to enable the technical means of the application to be more clearly understood, and to be implemented in accordance with the content of the description, and in order to enable the above and other purposes, characteristics and advantages of the application to be more apparent and easy to understand, the following specific embodiments of the application are described. BRIEF DESCRIPTION OF DRAWINGS

[0041] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are intended to only illustrate preferred embodiments and are not considered limiting of the application. Moreover, like reference numerals denote like parts throughout the several views in the drawings. In the drawings:

[0042] Figure 1 A flowchart of a Parkinson's disease treatment effect prediction method based on a multi-modal image model according to an embodiment of the present application is shown.

[0043] Figure 2 A framework diagram of a Parkinson's disease treatment effect prediction device based on a multi-modal image model according to an embodiment of the present application is shown.

[0044] Figure 3 A structural diagram of a computer device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0045] Exemplary embodiments of the present application will be described in detail with reference to the drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thoroughly and completely understood, and will fully convey the scope of the application to those skilled in the art.

[0046] Figure 1 A flowchart of a Parkinson's disease treatment effect prediction method based on a multi-modal image model according to an embodiment of the present application is shown. Specifically, as shown in Figure 1 the following steps are included:

[0047] Step S101, acquiring multi-modal vectors of Parkinson's disease patients at different treatment stages, wherein the multi-modal vectors include baseline data, follow-up PET / MR images, and corresponding contemporaneous clinical treatment schemes and multi-dimensional therapeutic effect evaluation index sequences.

[0048] In this embodiment, the baseline data contains the initial clinical status of the patient, such as medical history, demographic information, diagnostic information, first UPDRS score, etc., which is the starting point for evaluating disease progression and treatment effect. By acquiring PET and MR images at different treatment stages, the biological changes of Parkinson's disease in structure and function are dynamically tracked. Among them, the PET image provides molecular level information such as metabolism, neurotransmitter system function, inflammation, etc., such as dopamine transporter (DAT) function, dopamine D2 / D3 receptor density, brain glucose metabolism, etc., which are important biomarkers reflecting PD pathophysiological progression and drug action targets. Through PET, the activity changes of key regions such as the striatum and the substantia nigra can be quantified. MR images provide high-resolution brain structure information, such as brain atrophy, white matter lesions, morphological and microstructural changes of specific nuclei (such as the substantia nigra compact part), which can comprehensively depict the evolution of the disease from both structure and function dimensions. The contemporaneous clinical treatment plan records the detailed treatment plan (including drug types, doses, dosing frequencies, surgical parameters) received by the patient during image acquisition, and the relationship between specific treatment plans and images and clinical indicators is learned through modeling to predict the potential effects of different treatment strategies. For example, for the dose adjustment of levodopa, corresponding changes will occur in clinical manifestations and imaging. In addition to the standard UPDRS score, cognitive function assessment (such as MOCA), quality of life scale (such as PDQ-39), sleep assessment (such as PSQI), and depression and anxiety scale are also included, making the prediction more close to the actual benefits of the patient.

[0049] In step S102, the multi-modal vector is aligned on the time axis to construct a multi-modal and multi-time point time point data element sample sequence; a double-flow cross-attention encoder is deployed for the sample sequence in each time point data element, the features of one modality are projected into the feature space of another modality through the double-flow cross-attention encoder, and a fused multi-modal feature is output; wherein each PET flow / MR flow adopts an independent 3D convolutional neural network.

[0050] In this embodiment, PET and MR images have different physical principles, resolutions, contrasts, and biological information reflected (PET focuses on metabolism / function, and MR focuses on anatomy / structure), and simple splicing features often have poor results. The double-flow cross-attention encoder allows the features of one modality (such as the fine anatomical structure of MR) to guide or enhance the features of another modality (such as the functional metabolic information of PET), achieving complementarity rather than simple superposition, and being able to learn the deep correlation and synergistic effect between modalities and understand how the morphological changes of the substantia nigra cooperate with metabolic dysfunction, providing a more comprehensive perspective for disease progression and treatment response. In Parkinson's disease, the volume of the substantia nigra compact part and the uptake pattern of the striatal dopamine transporter (DAT) are key imaging markers. 3D CNN can better identify and quantify the above three-dimensional structural and functional information.

[0051] In an alternative way, the multi-modal vectors are aligned on the time axis by using a wavelet dynamic time warping method based on harmonic analysis, wherein the cost function of the wavelet dynamic time warping method is:

[0052]

[0053] wherein, is the cost between the ith and jth time series; is the s-layer wavelet decomposition operator; is the s-layer scale weight; is the phase feature of the treatment response function; is the phase integral constraint coefficient; , are the original data of the ith and jth samples, respectively.

[0054] In this embodiment, the treatment effect, image features and clinical indicators of Parkinson's disease often show different change patterns (e.g., short-term fluctuations, long-term trends) on different time scales. By using the wavelet decomposition operator to decompose the original time series x and y into sub-bands of different scales (frequencies), the above multi-scale features can be captured, and the alignment process can be more refined. In the treatment of Parkinson's disease, the patient's response to medication (such as fluctuations in "on period" and "off period") has periodicity and phase characteristics. By introducing phase information into alignment, the alignment result not only considers the amplitude similarity of the features, but also considers the consistency of the "rhythm" or "pace" in time. By constraining the phase integral to guide the alignment path and avoid points with serious misalignment in the treatment response phase, an alignment result that is more consistent with the medical practical significance can be obtained. For example, even if the amplitudes of the UPDRS scores of two patients are similar, if one is at the peak of medication and the other is at the trough of medication, the traditional DTW may align them, but the introduction of phase information can identify the above-mentioned internal phase difference.

[0055] In an alternative way, the calculation formula of the cross-attention weight matrix of the dual-flow cross-attention encoder is:

[0056]

[0057] wherein, is the query matrix of the PET modality; is the key matrix of the MR modality; is the query vector dimension; is the activation function; is the quantum fluctuation adjustment coefficient; is the feature matrix of the PET modality; is a feature matrix of the MR modality; is a super-symmetry permutation matrix; is a matrix trace operator.

[0058] In this embodiment, quantum fluctuation regulation is a kind of small, random disturbance or "noise", which is similar to simulating quantum fluctuations in physical systems, and helps the model to explore a wider parameter space during the training process. Traditional attention only focuses on the matching of explicit features, while ignoring the potential and nonlinear correlations between modalities. By the trace of the feature matrix and the super-symmetry permutation matrix, the complex interaction between the features of the PET and MR modalities is captured at a higher level, not just a simple dot product similarity, which helps the model to discover hidden complementary information from seemingly unrelated features. Super-symmetry theory is used in physics to describe the symmetry between bosons and fermions, which can capture the deep, non-trivial structural correlations between different types of entities (features of PET and MR modalities). Through the super-symmetry permutation matrix, the model not only calculates the similarity between the PET query and the MR key, but also tries to understand and exchange information between the two modalities at a higher dimension or more abstract level, so that the fused features not only contain functional information but also have accurate spatial positioning, thereby improving the accuracy of downstream prediction.

[0059] In step S103, the fused multi-modal features are connected with the corresponding digital clinical treatment scheme vector of the time point to form a time point comprehensive feature vector; the time point comprehensive feature vector is input into a gated recurrent unit for time series information aggregation, and a final time point aggregation feature is output; the final time point aggregation feature is input into a multi-task adaptive prediction head to predict the UPDRS total score or specific subscale score of the patient at a future preset time point.

[0060] In this embodiment, the gated recurrent unit (GRU) can effectively capture the long-term dependence and dynamic evolution pattern in the time series data, so as to more accurately understand the progress and maintenance of the treatment effect. The UPDRS total score and the specific subscale score (such as the motor score and the non-motor score) reflect different dimensions of the disease. Through shared representation and mutual promotion between tasks, the overall prediction performance is improved, and the weight is dynamically adjusted according to the task importance, data characteristics, etc., to improve the generalization ability. Taking the treatment scheme as the input feature, the model not only predicts the natural progression of the disease, but also predicts the disease progression under a specific treatment scheme, which has direct guiding significance for doctors to individualize treatment strategies and optimize drug regimens, and realizes the transformation from prediction to decision support.

[0061] In an optional manner, the gated recurrent unit for time series information aggregation adopts a fractional calculus enhancement method, and the update rule of the hidden state is:

[0062]

[0063] wherein, is the hidden state at time t; is the input at time t-k; is the calculation result of the gated recurrent unit on the input ; is the gamma function; is the fractional derivative order; is the generalized binomial coefficient; is the historical time step.

[0064] In this embodiment, the fractional derivative order allows the model to explore in the continuous order space, so as to more accurately model the non-linear and non-local characteristics of disease progression, such as the slow trend of change or sudden phase change of the UPDRS score of a Parkinson's disease patient. In the prediction of the treatment effect of Parkinson's disease, different values of α correspond to different "memory" modes of disease progression or treatment response, providing an explanation space for the behavior of the model, which helps to understand which historical information in which period is more important for predicting the current state.

[0065] In an optional manner, the multi-task adaptive prediction head comprises a plurality of independent prediction branches;

[0066] wherein each prediction branch is used to predict one of the UPDRS total score or the specific subscale score;

[0067] The multi-task adaptive prediction head dynamically adjusts the weights of the respective prediction branches according to the importance of different prediction tasks, the distribution characteristics of the training data, and the performance of the model in historical prediction.

[0068] In this embodiment, the evaluation of the treatment effect of Parkinson's disease is multi-dimensional (UPDRS total score and multiple subscale scores). Multi-task learning allows the model to simultaneously learn shared and unique information between different prediction tasks. At the same time, when the amount of data of some tasks is small or the noise is large, the information of other related tasks can play a role in regularization. Compared with training multiple independent single-task models, multi-task learning avoids the dispersion and redundancy of knowledge by sharing the underlying features. The model does not need to learn the features of each task from scratch.

[0069] In an optional manner, before predicting the UPDRS total score or the specific subscale score of the Parkinson's disease patient at a future preset time point, the method further comprises:

[0070] For PET images, motion correction, attenuation correction, scatter correction, and standardization to a common template space are performed to eliminate individual differences and scanning artifacts;

[0071] For MR images, intensity inhomogeneity correction, skull stripping and registration to PET image space are performed to ensure accurate co-registration of PET and MR images.

[0072] In this embodiment, during PET scanning, radioactive particles will decay and scatter in the tissue, resulting in signal loss or false counting. Through attenuation correction and scatter correction, the true radioactive distribution information is recovered to ensure the accurate reflection of tracer uptake. The PET images of different patients are registered to a unified template space (such as M.N.I. space) to eliminate individual brain morphology differences, so that the image data between different patients can be directly compared and analyzed. The MR image (usually used to provide high-resolution anatomical information) is accurately registered to the PET image space to achieve perfect fusion of multi-modal images. The model can simultaneously use the functional information of PET and the structural information of MR to more comprehensively understand the pathophysiological changes.

[0073] In step S104, the UPDRS total score or specific sub-scale score is optimized by minimizing the Wasserstein distance between the predicted trajectory and the actual trajectory, and the contribution of the final time point aggregated feature to the final prediction result is quantified by the SHAP or LIME method.

[0074] In this embodiment, the UPDRS score of the patient in the Parkinson's disease treatment effect prediction of such complex time series data may present nonlinearity, volatility, and even plateau or deterioration in different treatment stages. MSE / MAE will be too large when the predicted trajectory and the actual trajectory have "displacement" but similar shape, leading to overfitting of noise or poor performance in actual application. The Wasserstein distance (also known as "earth moving distance") can measure the minimum "transportation" cost between two distributions, i.e. the minimum cost required to move the "mass" of one distribution to another. In the context of predicted trajectory and actual trajectory, it can more robustly measure the similarity of two curves in shape, position and trend. Even if the predicted trajectory deviates from the actual trajectory at a certain time point, as long as the overall trend and relative position relationship are close, the Wasserstein distance can give a smaller penalty, thereby guiding the model to learn the prediction ability that is more consistent with the actual disease progression. By optimizing the Wasserstein distance, the inherent distribution characteristics of the data can be better understood and learned, and the sensitivity to single time point noise can be reduced. SHAP based on Shapley value in game theory can fairly allocate the contribution of each feature to the prediction result, and reveal the gain or reduction effect of each input feature (such as PET image feature, MR image feature, clinical treatment scheme vector, baseline data, multi-dimensional efficacy evaluation index, etc.) on the final UPDRS score prediction. For example, through SHAP, the specific influence of factors such as striatal FDG-PET uptake, specific gene expression, drug type and dose, patient age, etc. on the prediction of treatment effect can be quantified. LIME constructs an interpretable simple model (such as a linear model) in the local area to approximate the behavior of a complex model, thereby explaining a single prediction, for example, for a specific patient, why the predicted UPDRS will decrease or increase and pointing out the most important influencing factors.

[0075] In an alternative way, the method further comprises:

[0076] Segmenting and measuring the volume of the substantia nigra, striatum and cortex of the PET / MR image to extract morphological, functional and metabolic quantitative features of the segmented regions.

[0077] In this embodiment, the volume measurement directly reflects the atrophy or structural changes of brain regions, which is an important indicator of neurodegenerative diseases. For example, the volume change of the substantia nigra may be related to the degree of loss of dopaminergic neurons. PET images reflect functional information such as neurotransmitter receptor density and metabolic activity in specific brain regions. For example, FDG-PET uptake in the striatum can reflect glucose metabolism levels, which are related to the function of the dopaminergic pathway; and specific tracer PET directly reflects dopamine synthesis capacity or receptor density, providing direct evidence for evaluating the impact of treatment on the dopamine system. Metabolic characteristics (PET) can also directly provide quantitative data on brain region metabolism, such as glucose metabolic rate reflected by FDG-PET, which is an important indicator of neuronal activity.

[0078] In an alternative way, the method further comprises:

[0079] Collecting tremor frequency and gait stability data of the patient through the mobile application to construct the time-series PRO data; wherein the gait stability data includes gait speed, stride length, and gait bilateral symmetry;

[0080] Matching the time-series PRO data with the contemporaneous PET / MR image features on the time axis.

[0081] In this embodiment, traditional PET / MR image examination is usually intermittent and cannot capture the symptom fluctuations of the patient between two examinations. The mobile application enables high-frequency and continuous data collection, for example, the patient uses the App to record tremor and gait data every day or every week, which can better reflect the dynamic changes of disease progression and treatment effect, make up for the lack of time resolution of traditional image data, and help to find subtle trends of symptom improvement or deterioration.

[0082] According to the scheme provided by the application, a multi-modal vector of a Parkinson's disease patient at different treatment stages is obtained, wherein the multi-modal vector includes baseline data, follow-up PET / MR images, and corresponding contemporaneous clinical treatment schemes and multi-dimensional efficacy evaluation index sequences; the multi-modal vector is aligned on a time axis to construct a time point data element sample sequence of multi-modal and multi-time points; a double-flow cross-attention encoder is deployed for each sample sequence in each time point data element, the double-flow cross-attention encoder is used to project the features of one modality to the feature space of another modality, and fused multi-modal features are output; wherein an independent 3D convolutional neural network is used for each PET flow / MR flow; the fused multi-modal features are connected with the corresponding digitalized clinical treatment scheme vector of the corresponding time point to form a time point comprehensive feature vector; the time point comprehensive feature vector is input into a gated recurrent unit of time sequence information aggregation, and a final time point aggregation feature is output; the final time point aggregation feature is input into a multi-task adaptive prediction head to predict the UPDRS total score or specific subscale score of the patient at a future preset time point; the UPDRS total score or specific subscale score is optimized by minimizing the Wasserstein distance between the predicted trajectory and the actual trajectory, and the contribution of the final time point aggregation feature to the final prediction result is quantified by the SHAP or LIME method. The application improves the accuracy and interpretability of Parkinson's disease treatment effect prediction by performing time sequence modeling on multi-modal and multi-time point data. Specifically, by aligning the multi-modal data on the time axis to construct a time point data element sample sequence, and using a gated recurrent unit to aggregate time sequence information, the dynamic trajectory of disease development and treatment effect can be captured. By minimizing the Wasserstein distance between the predicted trajectory and the actual trajectory, the actual disease progression path is more accurately fitted, which has important clinical significance for long-term treatment management. By predicting the UPDRS total score or specific subscale score of the patient at a future preset time point, the doctor is provided with a prospective treatment effect evaluation, which helps the doctor to adjust the treatment scheme in a timely manner according to the specific condition of the patient and the expected efficacy.

[0083] Figure 2 A framework schematic diagram of a Parkinson's disease treatment effect prediction device based on a multi-modal image model of an embodiment of the application is shown. The Parkinson's disease treatment effect prediction device based on the multi-modal image model includes:

[0084] A data acquisition module 210 is configured to acquire a multi-modal vector of a Parkinson's disease patient at different treatment stages, wherein the multi-modal vector includes baseline data, follow-up PET / MR images, and corresponding contemporaneous clinical treatment schemes and multi-dimensional efficacy evaluation index sequences;

[0085] The multimodal fusion module 220 is used to align the multimodal vectors on the time axis to construct a multimodal and multi-time point time point data element sample sequence; a dual-stream cross-attention encoder is deployed in the sample sequence of each time point data element, and the features of one modality are projected into the feature space of another modality through the dual-stream cross-attention encoder to output fused multimodal features; wherein, each PET stream / MR stream adopts an independent 3D convolutional neural network;

[0086] The temporal feature aggregation and prediction module 230 is used to connect the fused multimodal features with the digital clinical treatment plan vector corresponding to the corresponding time point to form a time point comprehensive feature vector; input the time point comprehensive feature vector into the gated loop unit of temporal information aggregation, and output the final time point aggregated feature; input the final time point aggregated feature into the multi-task adaptive prediction head to predict the patient's UPDRS total score or specific subscale score at a future preset time point;

[0087] The prediction optimization and interpretability analysis module 240 is used to optimize the UPDRS total score or specific subscale score by minimizing the Wasserstein distance between the predicted trajectory and the actual trajectory, and to quantify the contribution of the final time point aggregated features to the final prediction result by using SHAP or LIME methods.

[0088] Figure 3 The diagram shows a structural schematic of an embodiment of the computer device of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computer device.

[0089] like Figure 3 As shown, the computer device may include: a processor 302, a communications interface 304, a memory 306, and a communications bus 308.

[0090] The processor 302, communication interface 304, and memory 306 communicate with each other via communication bus 308. Communication interface 304 is used to communicate with other network elements such as clients or other servers. The processor 302 executes program 310, specifically performing the relevant steps in the above-described embodiment of the Parkinson's disease treatment effect prediction method based on a multimodal image model.

[0091] Specifically, program 310 may include program code that includes computer operation instructions.

[0092] The processor 302 can be a central processing unit, CPU, or an application-specific integrated circuit, ASIC, or one or more integrated circuits configured to perform the functions of an embodiment of the application. The computer device can include one or more processors of the same type or different types, such as one or more CPUs and one or more ASICs.

[0093] The memory 306 stores a program 310. The memory 306 can include both a volatile RAM memory and a non-volatile memory, such as at least one disk memory.

[0094] According to the scheme provided by the application, a multi-modal vector of a Parkinson's disease patient at different treatment stages is obtained, wherein the multi-modal vector includes baseline data, follow-up PET / MR images, and corresponding contemporaneous clinical treatment schemes and multi-dimensional therapeutic effect evaluation index sequences; the multi-modal vector is aligned on a time axis to construct a multi-modal and multi-time point time point data element sample sequence; a double-flow cross-attention encoder is deployed for each sample sequence in the time point data element, the double-flow cross-attention encoder is used to project the features of one modality to the feature space of another modality, and fused multi-modal features are output; wherein each PET flow / MR flow adopts an independent 3D convolutional neural network; the fused multi-modal features are connected with the corresponding digitalized clinical treatment scheme vector of the corresponding time point to form a time point comprehensive feature vector; the time point comprehensive feature vector is input into a gated recurrent unit of time sequence information aggregation, and a final time point aggregated feature is output; the final time point aggregated feature is input into a multi-task adaptive prediction head to predict the UPDRS total score or specific subscale score of the patient at a future preset time point; the UPDRS total score or specific subscale score is optimized by minimizing the Wasserstein distance between the predicted trajectory and the actual trajectory, and the contribution of the final time point aggregated feature to the final prediction result is quantified by SHAP or LIME method. The application improves the accuracy and interpretability of Parkinson's disease treatment effect prediction by time sequence modeling of multi-modal and multi-time point data. Specifically, by aligning the multi-modal data on the time axis to construct a time point data element sample sequence, and using a gated recurrent unit to aggregate time sequence information, the dynamic trajectory of disease development and treatment effect can be captured. By minimizing the Wasserstein distance between the predicted trajectory and the actual trajectory, the actual disease progression path is more accurately fitted, which has important clinical significance for long-term treatment management. By predicting the UPDRS total score or specific subscale score of the patient at a future preset time point, the doctor is provided with a prospective treatment effect evaluation, which helps the doctor to adjust the treatment scheme in a timely manner according to the specific condition of the patient and the expected therapeutic effect.

[0095] Those skilled in the art will appreciate that the modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and furthermore can be divided into multiple sub-modules or sub-units or sub-components. Any combination of all the features disclosed in the present specification (including the accompanying claims, abstract and drawings), and any method or apparatus so disclosed, can be taken in any combination, except that at least some of such features and / or processes or units are mutually exclusive, unless explicitly stated otherwise. Each feature disclosed in the present specification (including the accompanying claims, abstract and drawings) can be replaced by alternative features serving the same, equivalent or similar purpose, unless explicitly stated otherwise. Furthermore, the skilled person will appreciate that the combination of features of different embodiments implies that the features of the different embodiments are meant to be combined, unless explicitly stated otherwise. For example, in the claims below, any of the embodiments can be used in any combination. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unitary claim, several of the devices mentioned in the embodiments can be implemented by means of one and the same hardware item. The steps of the above-described embodiments, unless explicitly stated otherwise, are not to be understood as having to be carried out in the order in which they are described.

Claims

1. A method for predicting treatment effect of Parkinson's disease based on a multi-modal image model, characterized in that, The method comprises the following steps: acquiring a multi-modal vector of a Parkinson's disease patient at different treatment stages, wherein the multi-modal vector comprises baseline data, follow-up PET / MR images, and corresponding contemporaneous clinical treatment plans and multi-dimensional efficacy evaluation index sequences; aligning the multi-modal vector on a time axis to construct a multi-modal and multi-time point time point data element sample sequence; deploying a double-flow cross-attention encoder for each sample sequence in each time point data element, projecting the features of one modality to the feature space of another modality through the double-flow cross-attention encoder, and outputting fused multi-modal features; wherein each PET flow / MR flow adopts an independent 3D convolutional neural network; connecting the fused multi-modal features with the corresponding digitalized clinical treatment plan vector at the corresponding time point to form a time point comprehensive feature vector; inputting the time point comprehensive feature vector into a gated recurrent unit for time series information aggregation to output a final time point aggregated feature; and inputting the final time point aggregated feature into a multi-task adaptive prediction head to predict the UPDRS total score or specific subscale score of the patient at a future preset time point; optimizing the UPDRS total score or specific subscale score by minimizing the Wasserstein distance between the predicted trajectory and the actual trajectory, and quantifying the contribution of the final time point aggregated feature to the final prediction result by SHAP or LIME method.

2. The method for predicting the treatment effect of Parkinson's disease based on a multimodal image model according to claim 1, characterized in that, aligning the multi-modal vector on a time axis by using a wavelet dynamic time warping method based on harmonic analysis, wherein the cost function of the wavelet dynamic time warping method is: wherein, is the cost between the ith and jth time series; is the s-th layer wavelet decomposition operator; is the s-th layer scale weight; is the phase feature of the treatment response function; is the phase integral constraint coefficient; , are the original data of the ith and jth samples, respectively. 3.The method of claim 1, wherein the method further comprises: determining a treatment effect of the Parkinson's disease based on the predicted probability of the Parkinson's disease. The calculation formula of the cross-attention weight matrix of the double-flow cross-attention encoder is: wherein, is a query matrix for the PET modality; is a key matrix for the MR modality; is a query vector dimension; is an activation function; is a quantum fluctuation regulation coefficient; is a feature matrix for the PET modality; is a feature matrix for the MR modality; is a super-symmetric permutation matrix; is a matrix trace operator. 4.The method of claim 1, wherein the method further comprises: determining a treatment effect of the Parkinson's disease based on the predicted probability of the Parkinson's disease. The gated recurrent unit for time series information aggregation adopts a fractional calculus enhancement method, and the update rule of the hidden state thereof is: wherein, is the hidden state at time t; is the input at time t - k; is the result of the computation of the gated recurrent unit on the input ; is the gamma function; is the fractional derivative order; is the generalized binomial coefficient; is the history time step. 5.The method of claim 1, wherein the method further comprises: determining a treatment effect of the Parkinson's disease based on the predicted probability of the Parkinson's disease. The multi-task adaptive prediction head comprises a plurality of independent prediction branches; Each prediction branch is used to predict one of the UPDRS total score or specific subscale score. The multi-task adaptive prediction head dynamically adjusts the weights of each prediction branch according to the importance of different prediction tasks, the distribution characteristics of training data, and the performance of the model in historical prediction. 6.The method of claim 1, wherein the method further comprises: determining a treatment effect of the Parkinson's disease based on the predicted probability of the Parkinson's disease. The method further comprises: segmenting and measuring the volume of the substantia nigra, striatum, and cortex of the PET / MR image to extract morphological, functional, and metabolic quantitative features of the segmented regions. 7.The method of claim 1, wherein the method further comprises: determining a treatment effect of the PD therapy based on the predicted PD progression rate. Before predicting the UPDRS total score or specific subscale score of the Parkinson's disease patient at a future preset time point, the method further comprises: For PET images, motion correction, attenuation correction, scatter correction, and standardization to a common template space are performed to eliminate individual differences and scan artifacts; For MR images, intensity inhomogeneity correction, skull stripping, and registration to the PET image space are performed to ensure accurate alignment of the PET and MR images. 8.The method of claim 1, wherein the method further comprises: determining a treatment effect of the Parkinson's disease based on the predicted probability of the Parkinson's disease. The method further comprises: Collecting tremor frequency and gait stability data of the patient through a mobile application to construct time-series PRO data; wherein the gait stability data includes gait speed, stride length, and gait bilateral symmetry. The time-series PRO data is matched with contemporaneous PET / MR image features on a time axis. 9.A device for predicting treatment effect of Parkinson's disease based on a multi-modal image model, characterized in that, The method comprises the following steps: a data acquisition module is configured to acquire multi-modal vectors of Parkinson's disease patients at different treatment stages, wherein the multi-modal vectors include baseline data, follow-up PET / MR images, and corresponding contemporaneous clinical treatment plans and multi-dimensional efficacy evaluation index sequences; a multi-modal fusion module is configured to align the multi-modal vectors on a time axis, construct multi-modal and multi-time point time point data element sample sequences, deploy a double-flow cross-attention encoder for each sample sequence in each time point data element, project the features of one modality to the feature space of another modality through the double-flow cross-attention encoder, and output fused multi-modal features; wherein each PET flow / MR flow adopts an independent 3D convolutional neural network; a time-series feature aggregation and prediction module is configured to connect the fused multi-modal features with corresponding digitalized clinical treatment plan vectors at the corresponding time points to form time point comprehensive feature vectors, input the time point comprehensive feature vectors into a gated recurrent unit for time-series information aggregation, and output final time point aggregated features; and input the final time point aggregated features into a multi-task adaptive prediction head to predict the UPDRS total score or specific subscale score of the patient at a future preset time point; a prediction optimization and explainability analysis module is configured to optimize the UPDRS total score or specific subscale score by minimizing the Wasserstein distance between the predicted trajectory and the actual trajectory, and quantify the contribution of the final time point aggregated features to the final prediction result by SHAP or LIME methods.

10. A computer device comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction makes the processor execute the operations corresponding to the above-mentioned Parkinson's disease treatment effect prediction method based on a multi-modal image model.

Images