A method for constructing a dementia risk prediction model based on multi-modal data
Patent Information
- Application Number
- CN202611043265.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-14
- Publication Date
- 2026-09-29
AI Technical Summary
[0005]针对现有痴呆风险预测模型构建方法中存在的模态单一、特征工程不充分、泛化能力不足及可解释性差等问题，本发明提供了一种基于多模态数据、采用集成学习与嵌套交叉验证的痴呆风险预测模型的构建方法
[0024]（1）多模态融合，信息互补：融合认知行为、面部情绪、语音情绪和生理信号四类数据，各模态之间信息互补，显著提高了模型对痴呆的识别能力。特别是引入了情绪认知冲突相关的特征（如情绪不稳定指数等），填补了现有模型忽视情绪维度的空白。
Smart Images
Figure CN122842957A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical artificial intelligence technology, specifically relating to a method for constructing a dementia risk prediction model based on multimodal data. Background Technology
[0002] Dementia, especially Alzheimer's disease (AD), has become a major challenge in global public health. Mild cognitive impairment (MCI) is a transitional state between normal aging and dementia, often referred to as the "golden window" for dementia intervention. Accurate identification and intervention during the MCI stage can delay or even prevent the progression to dementia. Therefore, developing highly accurate dementia risk prediction models has significant clinical and social value.
[0003] Currently, the construction of dementia risk prediction models mainly faces the following technical problems: 1) Single data modality: Existing models mostly rely on single neuroimaging data (such as MRI, PET) or single neuropsychological scale data. Imaging data is costly and has poor accessibility, making it unsuitable for large-scale community screening; scale data is highly subjective and greatly influenced by the experience of the assessors. 2) Insufficient feature engineering: Existing methods have failed to fully explore the complementary information in multimodal data. In particular, emotion-cognitive interaction features (such as emotion-cognitive conflict) are generally ignored in the construction of existing models, while clinical research shows that these features are early sensitive indicators of MCI. 3) Insufficient model generalization ability: Most studies are based on single-center, small-sample data to build models. The generalization performance of the models on different populations and different data collection devices has not been fully verified, resulting in limited clinical application. 4) Poor interpretability: Although deep learning models have high accuracy, their "black box" nature makes it difficult for clinicians to understand the decision-making basis of the models, limiting their credibility and acceptance in medical scenarios.
[0004] Therefore, there is an urgent need in this field for a method to construct a dementia risk prediction model that can integrate multimodal behavioral data and physiological signals, and has good generalization ability and interpretability. Summary of the Invention
[0005] To address the problems of existing dementia risk prediction model construction methods, such as single modality, insufficient feature engineering, inadequate generalization ability, and poor interpretability, this invention provides a method for constructing a dementia risk prediction model based on multimodal data and employing ensemble learning and nested cross-validation.
[0006] To achieve the above objectives, the present invention employs the following technical solution:
[0007] This invention provides a method for constructing a dementia risk prediction model based on multimodal data, comprising the following steps:
[0008] S1: Obtain a clinical dataset containing multimodal raw data from healthy control subjects, subjects with mild cognitive impairment, and subjects with Alzheimer's disease; the multimodal raw data includes: facial image sequences, speech signals, physiological signals, and corresponding clinical diagnostic labels;
[0009] S2: Preprocess the multimodal raw data, the preprocessing including: facial image frame alignment and illumination normalization, speech signal denoising and endpoint detection, physiological signal filtering and artifact removal;
[0010] S3: Extract multidimensional feature vectors from the preprocessed multimodal data. The multidimensional feature vectors include: cognitive behavioral features, emotional features, and physiological features.
[0011] S4: Based on the multidimensional feature vectors extracted in step S3, construct an initial feature matrix and use a feature selection algorithm to select the optimal feature subset that is most correlated with the dementia state;
[0012] S5: Using the optimal feature subset and the corresponding clinical diagnostic labels as training data, an ensemble learning algorithm is used to train a dementia risk prediction model.
[0013] S6: The performance of the prediction model is evaluated using nested cross-validation, with the area under the curve, sensitivity, and specificity as evaluation indicators. The model parameters with the best performance are selected as the final prediction model.
[0014] Furthermore, the multimodal raw data in step S1 also includes: the subject's demographic information and neuropsychological scale scores; the demographic information includes age, gender, and years of education.
[0015] Furthermore, the cognitive behavioral features in step S3 include: semantic correctness labels for each test question, reaction delay time, coefficient of variation of reaction delay time, and accuracy decay slope; the semantic correctness labels are obtained by automatic speech recognition and semantic understanding of the speech signal.
[0016] Furthermore, the emotional features in step S3 include: time-series statistics of facial emotion effectiveness value, probability distribution of voice emotion categories, and emotion instability index; the facial emotion effectiveness value is obtained by extracting the spatiotemporal features of facial action units through a graph convolutional network and mapping them to a discrete emotion space; the emotion instability index is the ratio of the standard deviation of the emotion effectiveness value time-series curve to the mean of the absolute value of the first derivative.
[0017] Furthermore, the physiological characteristics in step S3 include: the baseline level of skin conductance response, the frequency and average amplitude of skin conductance response, the power spectral density of finger tremor, and heart rate variability parameters; the heart rate variability parameters include the ratio of low-frequency power to high-frequency power.
[0018] Furthermore, the feature selection algorithm in step S4 is a combination of feature importance ranking based on random forest and recursive feature elimination method; the selection criterion for the optimal feature subset is the smallest feature set that maximizes the area under the curve during cross-validation.
[0019] Furthermore, the ensemble learning algorithm in step S5 is any one of gradient boosting decision tree, random forest, or extreme gradient boosting; the training process employs cost-sensitive learning, assigning a higher misclassification cost weight to samples from subjects with mild cognitive impairment and Alzheimer's disease than to healthy controls.
[0020] Furthermore, the nested cross-validation in step S6 includes: outer cross-validation for evaluating the model's generalization performance, and inner cross-validation for hyperparameter optimization; the performance evaluation also includes calculating the calibration curve and Brill score to evaluate the degree of calibration of the model's predicted probabilities.
[0021] Furthermore, the method also includes step S7: using the Shapley additive interpretation method to calculate the contribution value of each feature to the prediction result, and outputting a personalized feature importance ranking chart for each subject.
[0022] Furthermore, the dementia risk prediction model constructed by the method is used to identify the probability that the subject belongs to a healthy control, mild cognitive impairment, or Alzheimer's disease; the model outputs a three-dimensional probability vector, with three components representing the predicted probability of belonging to a healthy control, mild cognitive impairment, and Alzheimer's disease, respectively.
[0023] Compared with the prior art, the present invention has the following advantages:
[0024] (1) Multimodal fusion and information complementarity: The model integrates four types of data: cognitive behavior, facial emotion, vocal emotion, and physiological signals. The information complements each modality, which significantly improves the model's ability to identify dementia. In particular, it introduces features related to emotional cognitive conflict (such as the emotional instability index), filling the gap in existing models that neglect the emotional dimension.
[0025] (2) Cost-sensitive learning improves clinical applicability: By assigning higher misclassification costs to MCI and AD samples, the model becomes more sensitive to diseases, reduces the false negative rate, and is more suitable for large-scale screening scenarios.
[0026] (3) Nested cross-validation to ensure generalization ability: The nested cross-validation framework separates model evaluation from hyperparameter optimization, effectively avoiding information leakage. The obtained model performance estimate is more realistic and reliable, and has good cross-center and cross-population generalization ability.
[0027] (4) Feature selection optimization to avoid overfitting: By combining random forest importance ranking with recursive feature elimination, the optimal feature subset is selected, and the feature dimension is reduced while ensuring prediction accuracy, thereby improving the robustness of the model.
[0028] (5) SHAP interpretability enhances clinical trust: The SHAP method provides explanations of feature contributions at both the global and individual levels, which helps clinicians understand the decision-making logic of the model and promotes the practical application of the model in clinical practice. Attached Figure Description
[0029] Figure 1 This is an overall flowchart of the method of the present invention. Detailed Implementation
[0030] To further illustrate the technical solution of the present invention, the present invention will be further described below through embodiments.
[0031] Example 1
[0032] like Figure 1 As shown in this embodiment, a method for constructing a dementia risk prediction model based on multimodal data includes the following steps:
[0033] S1. Obtain the clinical dataset:
[0034] Obtain multimodal raw data from at least 500 participants across different clinical settings. Participant categories include: healthy controls (HC), mild cognitive impairment (MCI), and Alzheimer's disease (AD). Each group has at least 100 participants.
[0035] Each participant completed a standardized testing procedure of approximately 15 minutes: Minutes 0-5: Resting baseline acquisition (watching a neutral landscape video); Minutes 5-15: Performing an emotional cognitive conflict test task (20 test items, including 10 emotional face Stroop tasks and 10 counterfactual emotional reasoning tasks).
[0036] Data acquisition equipment: A computer terminal equipped with a 1080p camera (30fps) and microphone array, and a mouse with an integrated GSR sensor (sampling rate 100Hz).
[0037] The multimodal raw data includes:
[0038] (1) Facial image sequence: Facial video of the subject during the standardized emotional cognitive conflict test task, with a sampling rate of ≥30fps;
[0039] (2) Speech signal: Speech recordings during the same task process, with a sampling rate ≥16kHz;
[0040] (3) Physiological signals: including skin conductance response (GSR) signals and finger tremor signals, with a sampling rate ≥100Hz;
[0041] (4) Clinical diagnostic label: The gold standard diagnosis (HC / MCI / AD) made by a neurologist based on internationally recognized diagnostic standards (such as NIA-AA standards).
[0042] (5) Demographic information: age, sex, years of education;
[0043] (6) Neuropsychological scale scores: MMSE, MoCA, NPI and other scale scores.
[0044] S2. Multimodal data preprocessing:
[0045] The above multimodal raw data undergoes preprocessing, including:
[0046] (1) Face image frame alignment and illumination normalization: Multi-task concatenated convolutional network (MTCNN) is used for face detection and alignment, and after alignment, the images are uniformly cropped to 112×112 pixels; histogram equalization is used for illumination normalization; and temporal smoothing is performed on consecutive frames to eliminate acquisition noise.
[0047] (2) Noise reduction and endpoint detection of speech signal: Spectral subtraction is used to suppress background noise; dual threshold method is used to detect endpoints and extract effective speech segments (average 1.2~2.5 seconds per segment); pre-emphasis is applied to the speech signal to enhance high-frequency components.
[0048] (3) Physiological signal filtering and artifact removal: A bandpass filter (0.05~2Hz) was used to filter the GSR signal to remove high-frequency noise and low-frequency drift; wavelet transform was used to remove motion artifacts in the finger tremor signal.
[0049] S3. Multi-dimensional feature extraction:
[0050] Extracting multidimensional feature vectors from preprocessed multimodal data mainly includes:
[0051] (1) Cognitive and behavioral characteristics:
[0052] By performing automatic speech recognition (ASR) and semantic understanding on the speech signal, the following features are extracted:
[0053] Semantic correctness tags : A binary correctness indicator for each test question;
[0054] Response delay time The time interval from stimulus presentation to the start of the subject's response;
[0055] Coefficient of variation of reaction delay time: This reflects the stability of the reaction time;
[0056] Accuracy decay slope: The test task is divided into segments according to time sequence, and the slope of the linear regression is fitted to show the change of accuracy with the duration of the task.
[0057] (2) Emotional characteristics:
[0058] Time-series statistics of facial emotion efficacy value: The spatiotemporal features of facial action units (AUs) are extracted using a graph convolutional network (GCN) and mapped to a discrete emotion space (Ekman's six basic emotions + neutral), outputting the emotion efficacy value. A frame-by-frame sequence of [-1,+1].
[0059] Extract the following statistics from the time series: mean, standard deviation, maximum value, minimum value, range, skewness, and kurtosis.
[0060] Probability distribution of voice emotion categories: Using a pre-trained voice emotion recognition model (such as a model based on VGGish or wav2vec2.0), the probability distribution of five emotion categories (neutral, happy, sad, fearful, and angry) is output. .
[0061] Emotional Instability Index: The ratio of the standard deviation of the emotional efficacy time series curve to the mean of the absolute value of its first derivative, i.e., the emotional instability index. It is used to quantify the volatility of emotions.
[0062] (3) Physiological characteristics:
[0063] Baseline level of skin conductance response (GSR) base : The average value of GSR in the resting state 1 minute before the test begins.
[0064] Frequency and average amplitude of skin conductance response (SCR): Identify SCR events within 1 to 4 seconds after stimulus presentation, and extract the SCR frequency (times / minute) and average SCR amplitude (μS).
[0065] Power spectral density and heart rate variability parameters of finger tremor: Fourier transform was performed on the finger tremor signal to extract the power spectral density (PSD), the energy distribution in the 0.5~12Hz frequency band was calculated, and features such as dominant frequency, spectral entropy, and power ratio were extracted.
[0066] Heart rate variability (HRV) parameters: extract the heart rate interval from the GSR signal (using the heart rate artifact characteristics of GSR), calculate the low-frequency power (LF, 0.04~0.15Hz), high-frequency power (HF, 0.15~0.4Hz), and the ratio of low-frequency power to high-frequency power, i.e., the LF / HF ratio.
[0067] S4. Optimal Feature Subset Selection
[0068] Based on the multidimensional feature vectors extracted in step S3, an initial feature matrix is constructed. Where N is the number of samples, M is the initial feature dimension, and a feature selection algorithm is used to select the optimal feature subset that is most relevant to the dementia state;
[0069] The feature selection algorithm is based on the feature importance ranking of Random Forest combined with the recursive feature elimination method. The Random Forest model is used to calculate the importance score of each feature and sort them in descending order of importance. The Recursive Feature Elimination (RFE) method is used to start from all features and remove the feature with the lowest importance in each round. The cross-validation AUC of the feature subset in each round is calculated.
[0070] The selection criterion for the optimal feature subset is: the smallest set of features that maximizes the area under the curve during cross-validation is considered the optimal feature subset. , where K is the selected optimal feature dimension.
[0071] This step can reduce feature redundancy, lower the risk of model overfitting, and improve model training efficiency and generalization ability.
[0072] S5. Prediction Model Training:
[0073] The optimal feature subset and its corresponding clinical diagnostic labels are used as training data, and an ensemble learning algorithm is used to train a dementia risk prediction model.
[0074] The ensemble learning algorithm is any one of gradient boosting decision tree, random forest, or extreme gradient boosting; the training process employs cost-sensitive learning, assigning a higher misclassification cost weight to samples from subjects with mild cognitive impairment and Alzheimer's disease than to healthy controls.
[0075] Specifically, this embodiment will use the optimal feature subset The model is trained using the corresponding clinical diagnostic labels y∈{0,1,2} (0=HC, 1=MCI, 2=AD) as training data and an ensemble learning algorithm.
[0076] A cost-sensitive ensemble learning strategy is adopted, as follows:
[0077] The XGBoost algorithm is preferred for ensemble learning, with the following cost-sensitive learning settings:
[0078] Healthy control (HC) sample weights =1.0;
[0079] Mild cognitive impairment (MCI) sample weights =α (α can be 2.0~5.0).
[0080] The principle is that the misclassification cost of MCI and AD samples is higher than that of HC samples, which makes the model pay more attention to abnormal samples during training and improve the detection rate (sensitivity) of diseases.
[0081] The objective function for model training is:
[0082] ;
[0083] in, Let cross-entropy be the loss function. These are regularization terms (including the number of trees, leaf node weights, etc.). The cost weight is the category to which sample i belongs.
[0084] S6. Model Performance Evaluation and Optimization:
[0085] Nested cross-validation was used to evaluate the performance of the prediction model, with the area under the curve, sensitivity, and specificity as evaluation metrics. The model parameters with the best performance were selected as the final prediction model.
[0086] Nested cross-validation includes:
[0087] The outer cross-validation layer is used to evaluate the model's generalization performance. It employs 5-fold cross-validation, dividing the dataset into five parts, alternating between using one part as the test set and the remaining four parts as the training set. The outer layer is used to evaluate the model's generalization performance.
[0088] Inner-layer cross-validation is used for hyperparameter optimization: within each outer-layer training set, 3-fold or 5-fold cross-validation is used again. Optimized hyperparameters include: learning rate, maximum tree depth, subsample ratio, and column sample ratio (colsample_bytree).
[0089] Evaluation metrics: Area under the curve (AUC): macro-average AUC and one-vs-rest AUC for each category; Sensitivity = TP / (TP+FN); Specificity = TN / (TN+FP); Calibration Curve and Brier Score: assess the consistency between the model's predicted probability and the actual observed probability, i.e., the degree of calibration of the predicted probability.
[0090] The model parameter combination with the highest average AUC in the outer cross-validation is selected as the final prediction model.
[0091] S7, Model Explanation
[0092] The Shapley Additive exPlanations (SHAP) method is used to calculate the contribution of each feature to the prediction results, and outputs a personalized feature importance ranking map for each subject. SHAP values are based on Shapley values in game theory and can quantitatively calculate the marginal contribution of each feature to the prediction results.
[0093] The specific steps are as follows:
[0094] For each sample in the test set, calculate the SHAP value of each feature;
[0095] Global level: Draw a bar chart of SHAP feature importance to show the average |SHAP value| of each feature across all samples;
[0096] At the individual level: Draw a SHAP force plot to show the prediction process of a single sample and the contribution direction (positive or negative contribution) of each feature.
[0097] This step enhances the interpretability of the model, enabling clinicians to understand the basis of the model's predictions and improving the model's credibility and acceptance in clinical settings.
[0098] S8, Output:
[0099] The final dementia risk prediction model outputs a three-dimensional probability vector, with three components representing the predicted probability of belonging to a healthy control, mild cognitive impairment, and Alzheimer's disease, respectively, which is used to identify the probability of a subject belonging to a healthy control, mild cognitive impairment, or Alzheimer's disease.
[0100] The model can receive multimodal data from a new subject and output a probability vector indicating whether the subject belongs to HC, MCI, or AD. ,satisfy .
[0101] Example 2
[0102] To verify the advantages of the multimodal fusion strategy of this invention, an ablation experiment was designed to compare the model performance under different modal combinations:
[0103] Table 1 Ablation Experiment Results
[0104] Demographics + Scales Only 0.76 0.68 0.79 Cognitive and behavioral characteristics only 0.82 0.75 0.83 Emotional characteristics only 0.79 0.71 0.81 Physiological characteristics only 0.74 0.66 0.78 Cognition + Emotion 0.88 0.83 0.89 Cognitive + Physiological 0.85 0.80 0.86 Emotions + Physiology 0.84 0.79 0.85 The full-modal fusion of the present invention 0.95 0.92 0.94
[0105] As shown in Table 1, the performance of a single modality is limited, while the performance of a dual-modality combination is significantly improved compared to a single modality. The full-modality fusion strategy of this invention (including cognitive behavior, facial emotions, voice emotions, and physiological signals) achieves the best performance, with an AUC of 0.95, which is significantly better than any single-modality or dual-modality combination.
[0106] Example 3
[0107] This embodiment compares the performance of different ensemble learning algorithms on the feature set of this invention:
[0108] Table 2 Performance results of different ensemble learning algorithms
[0109] Random Forest 0.91 8.2 0.5 Gradient boosting tree 0.93 15.6 0.8 XGBoost 0.95 6.5 0.3 LightGBM 0.94 4.2 0.2 Support Vector Machine 0.85 12.3 1.2
[0110] As shown in Table 2, XGBoost performs best in terms of AUC (0.95) and has fast training and inference speeds, making it suitable as a preferred embodiment of the present invention.
[0111] The foregoing has shown and described the main features and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.
[0112] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
Claims
1. A method for constructing a dementia risk prediction model based on multimodal data, characterized in that, Includes the following steps: S1: Obtain a clinical dataset containing multimodal raw data from healthy control subjects, subjects with mild cognitive impairment, and subjects with Alzheimer's disease; The multimodal raw data includes: facial image sequences, speech signals, physiological signals, and corresponding clinical diagnostic labels; S2: Preprocess the multimodal raw data, the preprocessing including: facial image frame alignment and illumination normalization, speech signal denoising and endpoint detection, physiological signal filtering and artifact removal; S3: Extract multidimensional feature vectors from the preprocessed multimodal data. The multidimensional feature vectors include: cognitive behavioral features, emotional features, and physiological features. S4: Based on the multidimensional feature vectors extracted in step S3, construct an initial feature matrix and use a feature selection algorithm to select the optimal feature subset that is most correlated with the dementia state; S5: Using the optimal feature subset and the corresponding clinical diagnostic labels as training data, an ensemble learning algorithm is used to train a dementia risk prediction model. S6: The performance of the prediction model is evaluated using nested cross-validation, with the area under the curve, sensitivity, and specificity as evaluation indicators. The model parameters with the best performance are selected as the final prediction model.
2. The method for constructing a dementia risk prediction model based on multimodal data according to claim 1, characterized in that, The multimodal raw data in step S1 also includes: the subject's demographic information and neuropsychological scale scores; the demographic information includes age, gender, and years of education.
3. The method for constructing a dementia risk prediction model based on multimodal data according to claim 1, characterized in that, The cognitive behavioral features in step S3 include: semantic correctness labels for each test question, reaction delay time, coefficient of variation of reaction delay time, and accuracy decay slope; the semantic correctness labels are obtained by automatic speech recognition and semantic understanding of the speech signal.
4. The method for constructing a dementia risk prediction model based on multimodal data according to claim 1, characterized in that, The emotional features in step S3 include: time-series statistics of facial emotion effectiveness value, probability distribution of voice emotion categories, and emotion instability index; the facial emotion effectiveness value is obtained by extracting the spatiotemporal features of facial action units through a graph convolutional network and mapping them to a discrete emotion space; the emotion instability index is the ratio of the standard deviation of the emotion effectiveness value time-series curve to the mean of the absolute value of the first derivative.
5. The method for constructing a dementia risk prediction model based on multimodal data according to claim 1, characterized in that, The physiological characteristics in step S3 include: the baseline level of skin conductance response, the frequency and average amplitude of skin conductance response, the power spectral density of finger tremor, and heart rate variability parameters; the heart rate variability parameters include the ratio of low-frequency power to high-frequency power.
6. The method for constructing a dementia risk prediction model based on multimodal data according to claim 1, characterized in that, The feature selection algorithm in step S4 is a combination of feature importance ranking based on random forest and recursive feature elimination method; the selection criterion for the optimal feature subset is the smallest feature set that maximizes the area under the curve during cross-validation.
7. The method for constructing a dementia risk prediction model based on multimodal data according to claim 1, characterized in that, The ensemble learning algorithm in step S5 is any one of gradient boosting decision tree, random forest, or extreme gradient boosting; the training process adopts cost-sensitive learning, assigning a higher misclassification cost weight to samples from subjects with mild cognitive impairment and Alzheimer's disease than to healthy controls.
8. The method for constructing a dementia risk prediction model based on multimodal data according to claim 1, characterized in that, The nested cross-validation in step S6 includes: outer cross-validation for evaluating model generalization performance and inner cross-validation for hyperparameter optimization; the performance evaluation also includes calculating calibration curves and Brill scores to evaluate the degree of calibration of the model's predicted probabilities.
9. The method for constructing a dementia risk prediction model based on multimodal data according to claim 1, characterized in that, The method further includes step S7: using the Shapley additive interpretation method to calculate the contribution value of each feature to the prediction result, and outputting a personalized feature importance ranking chart for each subject.
10. The method for constructing a dementia risk prediction model based on multimodal data according to claim 1, characterized in that, The dementia risk prediction model constructed by the method is used to identify the probability that a subject belongs to a healthy control, mild cognitive impairment, or Alzheimer's disease; the model outputs a three-dimensional probability vector, with three components representing the predicted probability of belonging to a healthy control, mild cognitive impairment, and Alzheimer's disease, respectively.