Parkinson's disease heterogeneity typing method based on multi-modal characteristics

Abstract

The invention discloses a Parkinson's disease heterogeneity typing method based on multi-modal features, which mainly comprises the steps of modal data acquisition, modal feature extraction and feature measurement and model construction, and comprises the following steps: acquiring three-dimensional information of voice information, facial expression information and eye movement information as original data in a modal data acquisition stage; compared with single information in the prior art, the accuracy of the typing result can be effectively improved, the typing result is more comprehensive and objective, and the typing result obtained through the scheme can serve as a useful reference for medical staff to make typing judgment.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence-assisted diagnosis, and in particular to a method for heterogeneous classification of Parkinson's disease based on multimodal features. Background Technology

[0002] Parkinson's disease (PD), Parkinsonian type of multiple system atrophy (MSA-P) and cerebellar type (MSA-C), and progressive supranuclear palsy (PSP) are all neurodegenerative diseases. They often present with similar symptoms in the early stages, such as bradykinesia, rigidity, and postural abnormalities, making clinical symptom-based classification and diagnosis challenging. Because these diseases differ significantly in their pathological basis, progression rate, and treatment response, accurate differentiation between healthy individuals (HC) and different disease types is crucial for early screening and clinical decision-making. Currently used MRI, DAT-SPECT, and some biomarkers can aid in disease identification, but they often lack sufficient sensitivity and specificity in the early stages, and are costly and require sophisticated equipment, making them unsuitable for widespread screening. With the development of behavioral neuroscience, numerous studies have found that multimodal behavioral signals such as facial expressions, speech, and eye movements can reflect early neurological functional impairment characteristics in different diseases. For example, PD often presents with reduced facial expressions and monotonous speech, MSA-C is accompanied by significant cerebellar instability, and PSP shows specific abnormalities in vertical eye movements and facial control. However, most existing technologies are based on a single modality, resulting in limited feature extraction and an inability to fully represent the damage patterns of diseases in different circuits of the central nervous system. Furthermore, while some deep learning methods can improve performance, their lack of interpretability and cross-scenario applicability limits their clinical application. Thus, existing models for assisting in the diagnosis of Parkinson's disease often rely on single-source data (such as facial, eye movement, or speech data), leading to inaccurate outputs and failing to provide effective diagnostic assistance to doctors. Therefore, there is an urgent need for a non-invasive diagnostic technology that can comprehensively utilize multimodal behavioral signals and combine appropriate machine learning and metric learning methods to achieve early identification of multiple disease types, thereby overcoming the shortcomings of existing methods in terms of sensitivity, specificity, and clinical interpretability. Summary of the Invention

[0003] To address the technical problem that existing auxiliary diagnostic models for Parkinson's disease use a relatively simple data source modality, resulting in poor accuracy of the output results, this invention provides a solution.

[0004] To achieve the above objectives, this invention provides a method for heterogeneous classification of Parkinson's disease based on multimodal features, comprising the following steps:

[0005] Modal data acquisition: Acquiring the subject's speech information, eye movement information, and facial expression information;

[0006] Modal feature extraction: The speech information, eye movement information, and facial expression information are deconstructed to obtain different corresponding speech element feature groups, eye movement element feature groups, and facial expression element feature groups; and the constituent elements in the speech element feature groups, eye movement element feature groups, and facial expression element feature groups are arranged according to a preset strategy to obtain the corresponding speech target feature set, eye movement target feature set, and facial expression target feature set.

[0007] Model construction: The speech target feature set, the eye movement target feature set, and the facial expression target feature set are subjected to MLKR learning to obtain the corresponding speech measurement model, eye movement measurement model, and facial expression measurement model; the speech measurement model, eye movement measurement model, and facial expression measurement model are mapped into the corresponding measurement space to obtain the distance of the required subtype disease in the measurement space, and Gaussian process classifiers are trained on the measurement space embedding of each modality to output the modality probability;

[0008] Weighted fusion and result output: Weights are assigned to the three preliminary output probabilities, and the probability results of the subtyped diseases are calculated through the fusion formula. The disease with the highest probability is output as the judgment result.

[0009] As an improvement of this application, when outputting the results, a label is formed on the output results using the SHAP model. The label represents the contribution of the speech information, the eye movement information, and the facial expression information to the results.

[0010] As an improvement to this application, the speech element feature group comprises: glottal features, phonation features, articulation features, prosodic features, phonological structure features, and deep acoustic features based on representation learning.

[0011] As an improvement of this application, the eye movement element feature group comprises: time-domain features, frequency-domain features, nonlinear dynamic features, autocorrelation features, trajectory smoothness features, curvature features, and bending energy features of eye movement trajectory extraction.

[0012] As an improvement of this application, the composition of the facial expression element feature group includes: facial expression features, facial expression weakening features, movement amplitude features, muscle stiffness features, and autonomous facial expression defect features.

[0013] As an improvement to this application, the speech information needs to be denoised, endpoint detected, and framed before it is deconstructed.

[0014] As an improvement to this application, before deconstructing the eye-tracking information, it is also necessary to perform interpolation repair, noise filtering, and coordinate correction.

[0015] As an improvement to this application, the preset strategy is one of AUC ranking, variance filtering, and multi-model fusion weights.

[0016] As an improvement to this application, the weight fusion and result output steps specifically include:

[0017] Weighting: Facial expression information is weighted as w1, eye movement information is weighted as w2, and voice information is weighted as w3, where w1, w2, w3 ≥ 0, and w1 + w2 + w3 = 1;

[0018] The fusion formula is:

[0019] For any category The fusion formula is:

[0020] ;

[0021] in, Let p be the set of categories representing {PD, MSA−P, MSA−C, PSP, HC}, with 5 categories; p is the probability function / classifier output. This is the speech modal feature vector; Eye-tracking modality input feature vector; This is the feature vector of facial expression modality;

[0022] The final probability function is:

[0023] ;

[0024] The final output is determined as follows:

[0025] ;

[0026] As an improvement to this application, the corresponding step for the metric space in the model construction step is as follows:

[0027] S1. Generate monitoring signal

[0028] S2. Calculate the Mahalanobis distance for each metric model using the following function:

[0029] ;

[0030] in, , Represents speech modality; Represents eye-tracking modality; Indicates facial expression modality; Let represent the metric matrix corresponding to the m-th mode, which is a symmetric positive semi-definite matrix; This is the metric projection matrix corresponding to the m-th mode; Let be the input feature vector of the i-th sample in the m-th modality; Let be the input feature vector of the j-th sample in the m-th modality;

[0031] S3. Calculate the MLKR Gaussian kernel weights using the following function:

[0032] ;

[0033] in, Represents the natural exponential function;

[0034] S4. Kernel regression prediction, using the following function:

[0035] ;

[0036] in, This represents the labels of all training samples except the i-th sample. According to kernel weights in the m-th mode The weighted sum of labels obtained by weighted summation; This represents the sum of kernel weights between sample i and the other samples in the m-th modality, excluding the i-th sample.

[0037] The optimization objectives for S5 and MLKR are defined by the following functions:

[0038] ;

[0039] in, , This is represented as the dimension of the m-th mode after MLKR learning, mapped to the corresponding target dimension after dimensionality reduction. The dimension before dimensionality reduction; This represents the metric projection matrix corresponding to the m-th mode, with dimension 1. ,in Let d represent the real number field, and d represent the original feature dimension. The target dimension of the m-th mode after MLKR learning is mapped to the corresponding metric space. Let be the dimension of the original feature of the m-th modality;

[0040] Iterative optimization yielded And based on Constructed symmetric positive semi-definite matrix The function used is:

[0041] ;

[0042] S6 and MLKR dimensionality reduction mappings use the following functions:

[0043] ;

[0044] in, The vector represents the dimension-reduced representation. Let be a dimensional real vector space, representing the feature space after dimensionality reduction; Let be the original input feature vector of the i-th sample in the m-th modality;

[0045] and Satisfying function:

[0046] ;

[0047] in, It is expressed as the difference between the metric space embedding vectors of sample i and sample j after dimensionality reduction in the m-th modality; (This is expressed as the difference between the original input feature vectors of sample i and sample j in the m-th mode). Represented as based on The constructed symmetric positive semi-definite matrix;

[0048] S7. Train GPC for each modality and output the probability:

[0049] S71: The index of the model is... ;

[0050] S72: The function used to generate the RBF kernel is:

[0051] ;

[0052] in, Indicates the first Signal variance parameters for each mode; For the first The length scale parameter of the modality;

[0053] S73: Output of the operation:

[0054] For new samples First, map: The function to output the class probability is:

[0055] .

[0056] The beneficial effects of this invention are as follows: Compared with the prior art, this invention provides a multimodal feature-based heterogeneous classification method for Parkinson's disease. The main steps include modal data acquisition, modal feature extraction, and feature measurement and model construction. By collecting information from three dimensions—voice information, facial expression information, and eye movement information—as raw data during the modal data acquisition stage, compared with the single information in the prior art, the accuracy of the classification results can be effectively improved, and the classification results are more comprehensive and objective. The classification results obtained by this scheme can serve as a useful reference for medical staff to make classification judgments. Attached Figure Description

[0057] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0058] To more clearly illustrate the present invention, the invention will be further described below with reference to the accompanying drawings.

[0059] In the following description, specific examples are given to provide a more in-depth understanding of the invention. It is obvious that the described embodiments are merely some, not all, of the embodiments of the invention. It should be understood that the specific embodiments described are for illustrative purposes only and are not intended to limit the scope of the invention.

[0060] It should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the said feature, integral, step, operation, element, or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, or combinations thereof.

[0061] To address the aforementioned technical problems, this application provides a method for heterogeneous classification of Parkinson's disease based on multimodal features. Please refer to the appendix. Figure 1 This includes the following steps:

[0062] A method for heterogeneous classification of Parkinson's disease based on multimodal features includes the following steps:

[0063] Modal data acquisition: Acquiring the subject's speech information, eye movement information, and facial expression information;

[0064] Modal feature extraction: The speech information, eye movement information, and facial expression information are deconstructed to obtain different corresponding speech element feature groups, eye movement element feature groups, and facial expression element feature groups; and the constituent elements in the speech element feature groups, eye movement element feature groups, and facial expression element feature groups are arranged according to a preset strategy to obtain the corresponding speech target feature set, eye movement target feature set, and facial expression target feature set.

[0065] Model construction: The speech target feature set, the eye movement target feature set, and the facial expression target feature set are subjected to MLKR learning to obtain the corresponding speech measurement model, eye movement measurement model, and facial expression measurement model; The speech measurement model, eye movement measurement model, and facial expression measurement model are mapped into the corresponding measurement space to obtain the distance of the required subtype disease in the measurement space, and Gaussian process classifiers are trained on the measurement space embedding of each modality to output the modality probability;

[0066] Weighted fusion and result output: Weights are assigned to the three preliminary output probabilities, and the probability results of the subtyped diseases are calculated through the fusion formula. The disease with the highest probability is output as the judgment result.

[0067] It is evident that by collecting information from three dimensions—voice, facial surface, and eyes—as raw data during the modal data acquisition stage, the accuracy of the classification results can be effectively improved compared to the single information used in existing technologies. The classification results are more comprehensive and objective, and the classification results obtained through this method can serve as a useful reference for medical personnel to make classification judgments.

[0068] In this embodiment, the SHAP model is used to label the output results, representing the contribution of speech, eye-tracking, and facial expression information structures. This is understandable; to improve the transparency of this method in practical applications, the SHAP model is used to generate analytical labels for the contributions, quantifying the influence of facial expression, speech, and eye-tracking information on the final judgment result, and presenting the final output in a visual or indexal form. This mechanism helps healthcare professionals with clinical backgrounds understand the association between multimodal features and different neurodegenerative diseases, thereby enhancing the credibility of the model results and its practical application value.

[0069] In this embodiment, the speech element feature group comprises: glottal features, vocalization features, articulation features, prosodic features, phonological structure features, and deep acoustic features based on representation learning; the eye movement element feature group comprises: time-domain features, frequency-domain features, nonlinear dynamic features, autocorrelation features extracted from eye movement trajectories, as well as trajectory smoothness features, curvature features, and bending energy features. It is worth noting that the bending energy mentioned in this application refers to an energy-type measure of the degree of trajectory curvature, essentially accumulating the "square of curvature" over the entire trajectory to quantify how "curved, shaky, and twisted" the trajectory is; the facial expression element feature group comprises... The features include: facial expression features, facial attenuation features, movement amplitude features, muscle stiffness features, and features related to spontaneous facial expression defects. Each of these features is highly representative within its corresponding feature group and significantly impacts the classification results. For deep acoustic features, these can be obtained from the Disvoice library, including but not limited to autoencoder bottleneck layer embedding features, band-level spectral reconstruction error features, and static deep acoustic descriptors based on statistical functions. Specifically, the autoencoder bottleneck layer embedding features are 256 features extracted from the bottleneck layer of a speech spectrogram encoded and decoded using a convolutional / recurrent autoencoder. A 128-dimensional latent embedding vector is used to characterize the high-order time-frequency structure of speech. The frequency band-level spectral reconstruction error feature is obtained by calculating the mean square error (MSE) between the autoencoder decoded spectrogram and the original input spectrogram across several frequency bands, resulting in a 128-dimensional frequency band reconstruction error vector used to capture disease-related abnormal spectral patterns and distortion distributions. The static deep acoustic descriptor based on statistical functions is obtained by calculating the mean, standard deviation, skewness, and kurtosis of the aforementioned basic deep acoustic descriptors (including bottleneck embedding and frequency band reconstruction error) over the entire speech segment, forming approximately 10... 3 A static depth acoustic feature vector is used to stably characterize the distribution characteristics of the patient's representation throughout the entire speech. It can be seen that in the multimodal classification framework of this application, depth acoustic features are mapped together with other modal features such as eye movement, gait, and posture to the corresponding metric space, providing supplementary information about the "speech motor control pathway" in the Gaussian process classification model based on Mahalanobis distance, thereby improving the probability prediction accuracy and overall classification performance of each subtype of PD, MSA-P, MSA-C, PSP, and HC, and enhancing the ability to characterize the heterogeneity of Parkinson's disease.

[0070] Specifically, during the sound acquisition process, participants completed 11 speech paradigms (SP01–SP11) in a fixed order, including sustained vowels, repeated consonants, alternating consonants, reading aloud, and tongue twisters. Sustained vowel paradigms included / a / , / o / , / i / , and / u / ; repeated consonants included papapa, tatata, and kakaka; alternating consonants were pataka; the reading aloud task included "The North Wind and the Sun"; and the tongue twister task included "Forty-four stone lions" and "A large flower bowl covered with a large flower toad." The speech was collected by a microphone and recorded at a sampling rate of 44.1 kHz or higher. Glottal features, vocal control features, articulation features, prosodic features, and phonological features were extracted. Furthermore, a deep representation learning network was used to extract bottleneck layer speech embeddings, achieving multi-dimensional speech behavior quantification. Additionally, denoising, endpoint detection, and frame segmentation were performed before feature extraction to reduce interference and improve the accuracy of the acquired features.

[0071] During the facial expression information acquisition phase, RGB cameras or corresponding devices were used to record facial videos of the subjects, covering four facial paradigms: natural expressions, active smiling, laughing, and active sadness. After face detection and correction, the videos were input into a large-scale facial expression visual model (including dfer-clip, emo-clip, and static image clip encoders), and semantic vectors were extracted by constructing specific cue word templates. The cue words included multi-dimensional observational semantics such as facial muscle activity, emotional expression, upper and lower face movements, eyelid movements, expression initiation delay, and stiffness (e.g., "Reduced facial expression / Non-reduced facial expression", "Flattened nasolabial folds / Prominent nasolabial folds", "Reduced corrugator activity / Normal corrugator activity", etc.). For each pair of cue words, a set of facial expression semantic distributions was generated through visual-text similarity calculation, forming high-dimensional facial expression feature vectors corresponding to each expression paradigm. This allows for the extraction of useful facial element features as the target feature set during subsequent deconstruction.

[0072] During the eye-tracking data acquisition phase, participants sequentially completed tasks such as 30° slow-motion tracking, forward target tracking (eye tracker + display screen), horizontal free saccades, and horizontal-vertical gaze. The eye tracker simultaneously acquired the gaze position sequence of both eyes. Basic temporal information such as position, velocity, and acceleration was extracted, and further calculated higher-order dynamic characteristics including root mean square value, peak-to-peak value, skewness, kurtosis, coefficient of variation, dominant frequency, bandwidth energy distribution, spectral entropy, energy operator, autocorrelation peak value, trajectory curvature, trajectory smoothness, bending energy, and mean jerk (the jerk of eye position over time, i.e., the first derivative of acceleration with respect to time). These indicators reflect neural functional states such as fixation stability, saccade amplitude control, cerebellar coordination, and tracking smoothness. In a further step, before feature extraction, the acquired eye-tracking trajectory underwent interpolation repair, noise filtering, and coordinate correction, similarly reducing the influence of interference terms and improving the accuracy of the acquired features.

[0073] In this embodiment, the preset strategy is one of AUC ranking, variance filtering, and multi-model fusion weights.

[0074] In this embodiment, the specific steps in the weight fusion and result output steps include:

[0075] Weighting: Facial expression information is weighted as w1, eye movement information as w2, and voice information as w3;

[0076] For any category The fusion formula is:

[0077] ;

[0078] Specifically, facial expression information has a weight of 0.5, eye movement information has a weight of 0.2, and voice information has a weight of 0.3; therefore, the fusion formula is:

[0079] ;

[0080] in, Let p be the set of categories representing {PD, MSA−P, MSA−C, PSP, HC}, with 5 categories; p is the probability function / classifier output. This is the speech modal feature vector; Eye-tracking modality input feature vector; The expression modality feature vector is defined as w1, w2, w3 ≥ 0, and w1 + w2 + w3 = 1. By fusing information from three complementary modalities, the misjudgment problem caused by noise, individual differences, or poor performance of a single modality in specific tasks is effectively overcome, and the accuracy of the final classification is significantly improved. The linear weighting method is easy to implement and deploy, and has been proven to achieve excellent results in practice.

[0081] The final probability function is:

[0082] ;

[0083] p(x) represents the probability distribution vector of sample x across all categories; p(c|x)]_(c∈C) represents the vector formed by calculating the corresponding probability p(c|x) for each category c in the category set C.

[0084] The final output is determined as follows:

[0085] ;

[0086] This represents the final disease classification result for sample x; This corresponds to finding the category c in the category set C that maximizes the probability p(c|x); and p(c|x) represents the probability that sample x belongs to category c. It can be seen that using the maximum probability principle for the final judgment is logically clear, the result is intuitive, and it is easy for medical staff to understand and adopt.

[0087] In a more specific implementation, the corresponding steps for the metric space in the model building process are as follows:

[0088] S1. Generate monitoring signal

[0089] S2. Calculate the Mahalanobis distance for each metric model using the following function:

[0090] ;

[0091] in, , Represents speech modality; Represents eye-tracking modality; Indicates facial expression modality; Let represent the metric matrix corresponding to the m-th mode, which is a symmetric positive semi-definite matrix; This is the metric projection matrix corresponding to the m-th mode; Let be the input feature vector of the i-th sample in the m-th modality; Let be the input feature vector of the j-th sample in the m-th modality; it is easy to understand that, unlike Euclidean distance, Mahalanobis distance uses a metric matrix. By considering the correlation and scale differences between features, a more discriminative distance metric for the task (i.e., disease classification) can be learned; metric matrix It is learned, which means that the model can automatically amplify the feature dimensions that are useful for distinguishing different disease subtypes, while suppressing the influence of noise or irrelevant features.

[0092] S3. Calculate the MLKR Gaussian kernel weights using the following function:

[0093] ;

[0094] in, This represents the natural exponential function; it's easy to understand that this formula transforms the previously calculated Mahalanobis distance into a Gaussian kernel weight. The closer the samples are, the larger their kernel weights are, indicating that they are more similar in features. The Gaussian kernel is a powerful non-linear tool that can capture complex similarity relationships in the feature space, and its exponential decay form ensures the smoothness of the similarity measurement and avoids the instability caused by hard thresholds.

[0095] S4. Kernel regression prediction, using the following function:

[0096] ;

[0097] in, This represents the labels of all training samples except the i-th sample. According to kernel weights in the m-th mode The weighted sum of labels obtained by weighted summation; This represents the sum of kernel weights between sample i and all other samples in the m-th modality, excluding the i-th sample. It's easy to understand that effective prediction using this formula relies directly on the local manifold structure of the training data. It assumes that similar patients should have similar diagnostic labels, which aligns with the basic logic of disease classification. This predicted value... With real labels The gap between them constitutes the objective of MLKR optimization and guides the learning of the metric matrix.

[0098] The optimization objectives for S5 and MLKR are defined by the following functions:

[0099] ;

[0100] in, , This is represented as the target dimension / dimension reduced to the corresponding metric space after the m-th mode is learned by MLKR. The dimension before dimensionality reduction; This represents the metric projection matrix corresponding to the m-th mode, with dimension 1. ,in represents the feature dimension after dimensionality reduction, and d represents the original feature dimension; The target dimension of the m-th mode after MLKR learning is mapped to the corresponding metric space. Let be the dimension of the original features of the m-th modality; it is easy to understand that this objective function directly links metric learning to the final classification and regression task, ensuring that the learned distance metric is task-oriented, that is, specifically optimized to improve disease typing performance, and by minimizing the prediction error, the metric space learned by the model can better generalize to unseen data.

[0101] In subsequent solutions, iterative optimization yielded... And based on Constructed symmetric positive semi-definite matrix The function used is:

[0102] ;

[0103] S6 and MLKR dimensionality reduction mappings use the following functions:

[0104] ;

[0105] in, The vector represents the dimension-reduced representation. Let be a dimensional real vector space, representing the feature space after dimensionality reduction; Let be the original input feature vector of the i-th sample in the m-th modality;

[0106] and Satisfying function:

[0107] ;

[0108] in, It is expressed as the difference between the metric space embedding vectors of sample i and sample j after dimensionality reduction in the m-th modality; (This is expressed as the difference between the original input feature vectors of sample i and sample j in the m-th mode). Represented as based on The constructed symmetric positive semi-definite matrix; in the above two formulas, the high-dimensional feature vectors are projected onto the low-dimensional learned embedding space, so that the Euclidean distance between samples is equal to the Mahalanobis distance learned in the original space; the projection process is a dimensionality reduction and feature selection process, which can remove redundant information and noise; in particular, through the above two formulas, the complex, modality-specific Mahalanobis distance is transformed into a simple, universal Euclidean distance; this allows the next step of mapping features of different modalities (speech, eye movement, facial expression) to the same metric space, that is, making them form embedding vectors z that can be directly compared, which greatly improves the robustness of the model and the accuracy of the output;

[0109] S7. Train GPC for each modality and output the probability:

[0110] S71: The index of the model is... ;

[0111] S72: The function used to generate the RBF kernel is:

[0112] ;

[0113] in, Indicates the first The signal variance parameter of each modality is used to control the amplitude range of the kernel function output, reflecting the distinguishing strength of the features in that modality for disease subtyping; For the first The length scale parameter of the modality controls the sensitivity of the kernel function to the distance between samples. A larger value indicates that the model is not sensitive to changes in distance and produces a smoother classification boundary. Represented as the first In this modality, the sample With sample The Gaussian process kernel function value between the two samples is used to measure the covariance or similarity between the two samples in the Gaussian process classifier. Indicates the first The first mode The feature representation vector of each sample after MLKR dimensionality reduction mapping; similarly... Indicates the first The first mode The feature representation vector of a sample after MLKR dimensionality reduction mapping; it can be seen that the kernel function accurately models the nonlinear similarity relationship between samples in the metric space, providing a mathematical basis for the Gaussian process classifier, and enabling the boundaries of different Parkinson's disease subtypes to be accurately characterized in the complex feature space.

[0114] S73: Output of the operation:

[0115] For new samples First, map: The function to output the class probability is:

[0116] ;

[0117] in, Let be the complete class probability distribution vector of sample x under the m-th modality: This is expressed as the conditional probability that sample x belongs to class c in the m-th modality; This represents a 5-dimensional real vector space. It is evident that the standard output form of the classifier for each modality (speech, eye movement, facial expression) uniformly represents the classification results of different modalities as a 5-dimensional probability vector, providing a standardized input for subsequent multimodal fusion. This formula allows the contribution of each modality to each disease category to be quantified and compared, providing basic data for the subsequent SHAP interpretability analysis module. It helps reveal the identification value of different modalities (speech, eye movement, facial expression) for specific disease subtypes, enabling clinicians to make more convincing output judgments.

[0118] The advantages of this invention are:

[0119] 1) The multimodal feature fusion method proposed in this invention can identify Parkinson's disease and its heterogeneous subtypes (including PD, MSA-P, MSA-C, PSP, and healthy controls) at an early stage without invasive examinations. The method simultaneously utilizes facial expression semantic features, speech acoustic features, and eye-tracking dynamics features to comprehensively reflect the functional state of the central nervous system in areas such as emotional expression, glottal control, articulation coordination, and eye-tracking accommodation, making the identification process more sensitive and reliable.

[0120] 2) Because this invention introduces a large-scale visual expression model to extract high-dimensional facial expression semantic embeddings, and combines the glottal, vocalization, articulation and prosodic features of speech, as well as the time domain, frequency domain, nonlinear dynamics and curvature related indicators of eye movement trajectory, the complementarity of multimodal information significantly improves the ability to capture subtle differences between different disease types, making the subsequent classification structure more accurate.

[0121] 3) This invention employs the MLKR method based on metric learning to map multimodal features to corresponding metric spaces, and combines it with a Gaussian process classification model using a Mahalanobis distance kernel, enabling the model to possess a robust distance metric structure and the ability to express uncertainty. This structure not only improves classification accuracy but also enhances the model's stability and generalization ability under small sample conditions.

[0122] 4) This invention also provides labels for analyzing contribution, quantifying the contribution of facial expressions, speech, and eye movement features in the final discrimination results, making the model highly interpretable, helping clinicians understand the correspondence between different modal features and disease pathological mechanisms, and improving the clinical credibility of the output results.

[0123] 5) This method has a standardized overall process and low dependence on equipment. It can be implemented under simple hardware conditions such as conventional photographic equipment, voice recording equipment and portable eye trackers. It is also easy to extend to remote monitoring and home follow-up scenarios, and has broad application value and promotion potential.

[0124] The above-disclosed embodiments are merely a few specific examples of the present invention, but the present invention is not limited thereto. Any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A method for heterogeneous classification of Parkinson's disease based on multimodal features, characterized in that, Includes the following steps: Modal data acquisition: Acquiring the subject's speech information, eye movement information, and facial expression information; Modal feature extraction: The speech information, eye movement information, and facial expression information are deconstructed to obtain different corresponding speech element feature groups, eye movement element feature groups, and facial expression element feature groups; and the constituent elements in the speech element feature groups, eye movement element feature groups, and facial expression element feature groups are arranged according to a preset strategy to obtain the corresponding speech target feature set, eye movement target feature set, and facial expression target feature set. Model construction: The speech target feature set, the eye movement target feature set, and the facial expression target feature set are subjected to MLKR learning to obtain the corresponding speech measurement model, eye movement measurement model, and facial expression measurement model; The speech measurement model, eye movement measurement model, and facial expression measurement model are mapped into the corresponding measurement space to obtain the distance of the required subtype disease in the measurement space, and Gaussian process classifiers are trained on the measurement space embedding of each modality to output the modality probability; Weighted fusion and result output: Weights are assigned to the three preliminary output probabilities, and the probability results of the subtyped diseases are calculated through the fusion formula. The disease with the highest probability is output as the judgment result.

2. The method for heterogeneous classification of Parkinson's disease based on multimodal features according to claim 1, characterized in that, When outputting the results, the SHAP model is used to generate labels for the output results, and the labels represent the contribution of the speech information, the eye movement information and the facial expression information to the results.

3. The method for heterogeneous classification of Parkinson's disease based on multimodal features according to claim 1, characterized in that, The speech element feature group consists of: glottal features, phonation features, articulation features, prosodic features, phonological structure features, and deep acoustic features based on representation learning.

4. The method for heterogeneous classification of Parkinson's disease based on multimodal features according to claim 1, characterized in that, The eye-tracking element feature set comprises: time-domain features, frequency-domain features, nonlinear dynamic features, autocorrelation features, as well as trajectory smoothness features, curvature features, and bending energy features.

5. The method for heterogeneous classification of Parkinson's disease based on multimodal features according to claim 1, characterized in that, The composition of the facial expression element feature group includes: facial expression features, facial expression weakening features, movement amplitude features, muscle stiffness features, and autonomous facial expression defect features.

6. The method for heterogeneous classification of Parkinson's disease based on multimodal features according to claim 1, characterized in that, Before deconstructing the speech information, it is necessary to perform noise reduction, endpoint detection, and frame segmentation.

7. The method for heterogeneous classification of Parkinson's disease based on multimodal features according to claim 1, characterized in that, Before deconstructing the eye-tracking information, it is necessary to perform interpolation repair, noise filtering, and coordinate correction.

8. The method for heterogeneous classification of Parkinson's disease based on multimodal features according to claim 1, characterized in that, The preset strategy is one of AUC ranking, variance filtering, and multi-model fusion weights.

9. The method for heterogeneous classification of Parkinson's disease based on multimodal features according to claim 1, characterized in that, The specific steps in the weight fusion and result output step include: Weighting: Facial expression information is weighted as w1, eye movement information as w2, and voice information as w3, where w1, w2, and w3 >= 0, and w1 + w2 + w3 = 1; The fusion formula is: For any category The fusion formula is: ； in, Let p be the set of categories representing {PD, MSA−P, MSA−C, PSP, HC}, with 5 categories; p is the probability function / classifier output. This is the speech modal feature vector; Eye-tracking modality input feature vector; This is the feature vector of facial expression modality; The final probability function is: ； The final output is determined as follows: 。 10. The method for heterogeneous classification of Parkinson's disease based on multimodal features according to claim 1, characterized in that, In the model construction steps, the corresponding steps for the metric space are as follows: S1. Generate monitoring signal ; S2. Calculate the Mahalanobis distance for each metric model using the following function: ； in, , Represents speech modality; Represents eye-tracking modality; Indicates facial expression modality; Let represent the metric matrix corresponding to the m-th mode, which is a symmetric positive semi-definite matrix; This is the metric projection matrix corresponding to the m-th mode; Let be the input feature vector of the i-th sample in the m-th modality; Let be the input feature vector of the j-th sample in the m-th modality; S3. Calculate the MLKR Gaussian kernel weights using the following function: ； in, Represents the natural exponential function; S4. Kernel regression prediction, using the following function: ； in, This represents the labels of all training samples except the i-th sample. According to kernel weights in the m-th mode The weighted sum of labels obtained by weighted summation; This represents the sum of kernel weights between sample i and the other samples in the m-th modality, excluding the i-th sample. The optimization objectives for S5 and MLKR are defined by the following functions: ； in, , This is represented as the dimension of the m-th mode after MLKR learning, mapped to the corresponding target dimension after dimensionality reduction. The dimension before dimensionality reduction; This represents the metric projection matrix corresponding to the m-th mode, with dimension 1. ,in Let d represent the real number field, and d represent the original feature dimension. The target dimension of the m-th mode after MLKR learning is mapped to the corresponding metric space. Let be the dimension of the original feature of the m-th modality; Iterative optimization yielded And based on Constructed symmetric positive semi-definite matrix The function used is: ； S6 and MLKR dimensionality reduction mappings use the following functions: ； in, The vector represents the dimension-reduced representation. Let be a dimensional real vector space, representing the feature space after dimensionality reduction; Let be the original input feature vector of the i-th sample in the m-th modality; and Satisfying function: ； in, It is expressed as the difference between the metric space embedding vectors of sample i and sample j after dimensionality reduction in the m-th modality; (This is expressed as the difference between the original input feature vectors of sample i and sample j in the m-th mode). Represented as based on The constructed symmetric positive semi-definite matrix; S7. Train GPC for each modality and output the probability: S71: The index of the model is... ; S72: The function used to generate the RBF kernel is: ； in, Indicates the first Signal variance parameters for each mode; For the first The length scale parameter of the modality; S73: Output of the operation: For new samples First, map: The function to output the class probability is: 。

Images