Depression risk dynamic assessment method and system based on multi-modal data fusion

Abstract

The invention discloses a depression risk dynamic assessment method and system based on multi-modal data fusion. The system comprises a multi-modal data acquisition module used for continuously acquiring physiological signals, behavior activities and original voice data of a user; the data preprocessing and feature engineering module is used for data cleaning and feature extraction; the multi-modal feature fusion module adopts a deep learning network based on an attention mechanism to perform adaptive weighted fusion on different modal features to generate deep fusion feature representation; the dynamic risk assessment model module analyzes the fusion features by using a pre-training model, and outputs the current depression risk level, probability and key influence factors; and the visual interaction and feedback module is used for result presentation and feedback collection. According to the method, through multi-dimensional data fusion and an attention mechanism, continuous, dynamic and personalized evaluation of the individual depression risk is realized, and the objectivity, comprehensiveness and accuracy of evaluation are improved.

Description

Technical Field

[0001] This invention relates to the field of mental health monitoring and digital healthcare technology, specifically to a method and system for dynamic assessment of depression risk based on multimodal data fusion. Background Technology

[0002] Depression is a common and serious mood disorder, and early identification and continuous monitoring are crucial for intervention and treatment. Traditional depression risk assessment mainly relies on regular clinical interviews or self-rating scales (such as the PHQ-9), which has limitations such as high subjectivity, assessment lag, and inability to reflect daily dynamic changes.

[0003] With the development of wearable devices, mobile computing, and artificial intelligence technologies, the use of passively collected data for mental health monitoring has become a research hotspot. Existing solutions mostly focus on the analysis of single-modal data, such as analyzing anxiety levels solely through heart rate variability (HRV) or inferring emotional states solely from smartphone usage patterns. These single-modal methods have limited information sources, their assessment results are easily influenced by random factors, and their generalization ability and robustness are insufficient. Furthermore, most systems only provide static or discontinuous assessments, lacking the ability to characterize and predict the continuous and dynamic evolution of an individual's depression risk.

[0004] Therefore, there is an urgent need for a technical solution that can comprehensively utilize multi-dimensional data, achieve adaptive feature fusion, and conduct dynamic and continuous risk assessment to provide more objective, comprehensive, and timely insights into psychological states. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the purpose of this invention is to provide a method and system for dynamic assessment of depression risk based on multimodal data fusion.

[0006] To achieve the aforementioned objective, the technical solution of the present invention is as follows: a dynamic assessment system, method, and storage medium for depression risk based on multimodal data fusion, which aims to achieve continuous, dynamic, and personalized assessment of individual depression risk by integrating multimodal data such as physiological, behavioral, and linguistic data.

[0007] In a first aspect, the present invention provides a dynamic assessment system for depression risk based on multimodal data fusion, comprising: The multimodal data acquisition module is used to continuously or periodically acquire multi-dimensional raw data from the user terminal. The multi-dimensional raw data includes at least physiological signal data, behavioral activity data, and language speech data. The data preprocessing and feature engineering module communicates with the multimodal data acquisition module to clean and standardize various types of raw data, and extract time-domain, frequency-domain and time-frequency-domain features related to depressive mood state to generate structured feature vectors. The multimodal feature fusion module, connected to the data preprocessing and feature engineering module, receives structured feature vectors and uses a deep learning fusion network based on an attention mechanism to dynamically learn and assign differentiated weights to features of different modalities, perform adaptive weighted fusion, and generate a unified deep fusion feature representation. The dynamic risk assessment model module, connected to the multimodal feature fusion module, has a built-in machine learning or deep learning model trained with labeled data. It is used to receive deep fusion feature representations, calculate and output the user's current depression risk level, risk probability score and key influencing factor analysis. The visualization interaction and feedback module, connected to the dynamic risk assessment model module, is used to visually present dynamic risk assessment results and historical risk trend charts to users and authorized third parties, and to provide personalized intervention suggestions and interactive feedback interfaces.

[0008] Preferably, the multimodal data acquisition module specifically includes: A physiological signal acquisition unit is used to acquire at least one of heart rate variability, skin conductance, and sleep quality data through a wearable device or a built-in sensor. A behavioral activity collection unit is used to collect at least one of the following data through mobile device sensors or logs: step count, screen usage time, social application activity, and diurnal activity rhythm data. The speech acquisition unit is used to acquire speech segments through an audio input device and extract at least one of the following: speech intonation features, language content semantic features, and dialogue interaction pattern features.

[0009] Preferably, the multimodal feature fusion module employs a deep learning fusion network based on an attention mechanism, comprising: Multiple parallel modality-specific encoding subnetworks are used to process the structured feature vectors of different modalities respectively, and learn the high-level abstract representation of each modality; A cross-modal attention fusion layer is used to calculate the correlation weights between the high-level abstract representations of each modality, and to perform weighted summation or concatenation based on the weights to generate a deep fusion feature representation.

[0010] Preferably, the dynamic risk assessment model module includes: The short-term state assessment sub-model is used to assess a user's mood fluctuations and depression risk status at the current moment or in the short term based on the fusion features within a recent time window. The long-term trend prediction sub-model is used to predict the development trend of a user's depression risk over a future period of time based on long-term historical fusion feature sequences. The system integrates short-term condition assessment results with long-term trend prediction results to generate a final dynamic risk assessment report.

[0011] Preferably, the system further includes: The model continuous learning and adaptation module connects with the dynamic risk assessment model module and the visualization interaction and feedback module. It is used to receive user feedback on assessment results and new labeled data, and to perform online incremental updates or periodic retraining of model parameters in the dynamic risk assessment model module, so as to realize personalized adaptation and continuous performance optimization of the model.

[0012] Secondly, this invention provides a dynamic assessment method for depression risk based on multimodal data fusion, applied to the aforementioned system, the method comprising: S1: Through the multimodal data acquisition module, continuously or periodically collect multidimensional raw data of the user's physiological signals, behavioral activities and speech. S2: Through the data preprocessing and feature engineering module, various types of raw data are cleaned, standardized, and feature extracted to generate structured feature vectors for each modality; S3: Through the multimodal feature fusion module, an attention-based fusion network is used to adaptively weight and fuse structured feature vectors to generate a deep fusion feature representation; S4: Through the dynamic risk assessment model module, the deep fusion feature representation is input into the pre-trained assessment model to calculate the user's current depression risk level, risk probability, and influencing factor analysis results; S5: Through the visualization interaction and feedback module, the evaluation results are output in a visual form, and the evaluation model is optimized based on user feedback.

[0013] Preferably, the adaptive weighted fusion in step S3 specifically involves: The importance score of each modality feature in a specific context is calculated using an attention mechanism. Based on importance scores, the feature representations of the corresponding modalities are dynamically weighted or selectively focused, and then fused, so that the modal features that contribute more to the current risk assessment receive higher weights.

[0014] Preferably, step S4 further includes: By combining users' historical fusion feature sequences and using time series analysis models or recurrent neural networks, dynamic change patterns of users' depression risk status can be identified, and the risk evolution trend can be reflected in the assessment results.

[0015] Preferably, the method further includes: S6: Based on user confirmation and correction feedback of historical assessment results, as well as newly added labeled data, the model in the dynamic risk assessment model module is incrementally learned to continuously optimize the accuracy of personalized risk assessment for the user.

[0016] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods described above.

[0017] The beneficial effects of this invention are reflected in: Comprehensiveness and objectivity: By integrating multimodal data such as physiological, behavioral, and linguistic data, the limitations of single-source data are overcome, providing a more comprehensive portrayal of psychological states and making the assessment results more objective.

[0018] Dynamic and continuous: Through continuous or periodic data collection and model analysis, dynamic monitoring and trend prediction of depression risk are achieved, enabling timely detection of subtle changes in the state.

[0019] Adaptability and accuracy: By introducing an attention mechanism for feature fusion, the importance of different data sources can be dynamically adjusted according to individual differences and specific contexts, thereby improving the effectiveness of feature representation and the accuracy of risk assessment.

[0020] Personalization and Evolution: Through user feedback and continuous learning mechanisms, the system can continuously adapt to the unique patterns of users, realize personalized optimization of the evaluation model, and improve the applicability and reliability of long-term use.

[0021] User-friendly interface and preventative measures: Visual interfaces and intervention suggestions enhance the system's usability, help users become more self-aware, and provide technical support for early warning and preventative intervention. Attached Figure Description

[0022] In the attached diagram: Figure 1 This is a structural block diagram of a dynamic assessment system for depression risk based on multimodal data fusion.

[0023] Figure 2 This is a schematic diagram of the attention mechanism fusion network for the multimodal feature fusion module.

[0024] Figure 3 This is a flowchart of a dynamic assessment method for depression risk based on multimodal data fusion.

[0025] Figure 4 This is a visual illustration of a dynamic risk assessment report. Detailed Implementation

[0026] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the invention, and not all of them. Unless otherwise specified, the embodiments and features described in this application can be combined with each other. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0027] Furthermore, "multiple" refers to two or more. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of a person skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the invention.

[0028] See Figure 1 The present invention provides a dynamic assessment system for depression risk based on multimodal data fusion, which mainly includes the following modules: Multimodal data acquisition module 101: This module is responsible for continuously or periodically acquiring raw data from smart terminals used by users on a daily basis (such as smartphones, smartwatches, dedicated audio devices, etc.).

[0029] Physiological signal acquisition unit: acquires heart rate and heart rate variability (HRV) data through photoplethysmography (PPG) sensors in smart bracelets / watches; acquires skin conductance activity data through a skin conductance response (GSR) sensor; and analyzes sleep quality data such as sleep duration and deep sleep ratio through accelerometers and algorithms.

[0030] Behavioral activity collection unit: Uses the phone's built-in accelerometer and gyroscope to record daily steps and activity intensity; obtains screen up time and application usage duration (especially social applications) through the operating system API; analyzes the time distribution of device usage and calculates characteristics such as work-rest regularity and circadian rhythm stability.

[0031] Speech Acquisition Unit: With user authorization and while ensuring privacy, short-term speech samples (such as daily mood diary readings) are periodically collected via a mobile phone microphone. Speech signal processing technology is used to extract acoustic features such as fundamental frequency, energy, and speech rate; natural language processing technology is used to perform sentiment analysis and semantic mining on the transcribed text (such as negative word frequency and first-person singular usage frequency); and interaction patterns such as response delay and utterance length in the dialogue are analyzed.

[0032] Data Preprocessing and Feature Engineering Module 102: This module receives raw data and processes it.

[0033] Data cleaning: removing outliers, handling missing signals (such as interpolation), and filtering noise.

[0034] Standardization: Normalizing data of different dimensions and ranges into a unified interval.

[0035] Feature extraction: Depression-related features are extracted for each data type. For example, time-domain features (SDNN, RMSSD) and frequency-domain features (LF, HF power) are extracted from HRV; daily / weekly activity variation rate and nighttime activity index are extracted from behavioral data; MFCC coefficient and intonation fluctuation are extracted from speech; and affective polarity score and lexical complexity are extracted from text. Finally, a structured feature vector is generated for each modality.

[0036] Multimodal feature fusion module 103: such as Figure 2 As shown, this module uses an attention-based neural network for fusion.

[0037] Modality coding subnetwork: The feature vector of each modality is input to an independent subnetwork (such as a fully connected layer or a small neural network) to learn a high-level representation of that modality. .

[0038] Cross-modal attention fusion layer: This layer computes a context vector and calculates the relevance of each modal representation to this context vector, obtaining attention weights α_i (e.g., through dot product or additive attention mechanisms). The weights reflect the importance of each modal information at the current evaluation time. Finally, the deep fusion feature representations are obtained. .

[0039] Dynamic Risk Assessment Model Module 104: This module receives... An evaluation will be conducted.

[0040] Short-term state assessment sub-model: Gradient boosting tree, support vector machine or shallow neural network can be used. Input the fused features of the most recent 24 hours or week, and output the current depression risk level (such as low, medium or high) and probability score.

[0041] Long-term trend prediction sub-model: Using recurrent neural networks such as LSTM and GRU, inputting fused feature sequences from the past few weeks or months, learning dynamic patterns, and predicting the risk trend (such as rising, flat, or falling) for the next week.

[0042] Comprehensive Analysis: The final report is generated by combining short-term assessment results and long-term trend predictions. The model can provide feature importance analysis, indicating whether physiological indicators (such as poor sleep), behavioral indicators (such as reduced social interaction), or linguistic indicators (such as monotone speech) contribute the most to the risk score at the current stage.

[0043] Visual interaction and feedback module 105: such as Figure 4 This module displays the results to the user through an app or web interface.

[0044] Visual presentation: The current risk level and probability are displayed in the form of a dashboard; historical risk trend changes are displayed in a line chart; and the magnitude of each modality's influencing factors is displayed in a radar chart or bar chart.

[0045] Interactive feedback: Provides a feedback option of "Assess whether this matches your current feelings?" (Yes / No). Offers personalized suggestions based on the results, such as "A recent decrease in activity level has been detected; we suggest trying a 15-minute walk daily."

[0046] Authorized access: With the user's consent, the desensitized trend report can be shared with a designated therapist or family member.

[0047] Model Continuous Learning and Adaptation Module 106 (Optional but Preferred): This module collects user feedback (such as corrections to assessment results) and potentially new labeled data (such as standard questionnaires filled out by users periodically). Using this new data, the model parameters in the Dynamic Risk Assessment Model Module 104 are fine-tuned through online learning or periodic batch training to better suit the user's specific patterns, achieving personalized accuracy improvement.

[0048] See Figure 3 A corresponding dynamic assessment method for depression risk based on multimodal data fusion, corresponding to the above system, includes the following steps: S301: Continuously collect multi-dimensional raw data.

[0049] S302: Perform data preprocessing and feature extraction to generate feature vectors for each modality.

[0050] S303: Adaptive weighted fusion of multimodal features using an attention mechanism network.

[0051] S304: Input the fusion features into the assessment model, calculate the current risk level and probability, and combine them with historical sequence analysis to analyze trends.

[0052] S305: Visualize the evaluation report and receive user feedback.

[0053] S306 (optional): Incrementally learn and optimize the evaluation model based on feedback and new data.

[0054] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the various steps of the above-described method embodiments.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0056] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0057] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A dynamic assessment system for depression risk based on multimodal data fusion, characterized in that, include: A multimodal data acquisition module is used to continuously or periodically acquire multi-dimensional raw data from a user terminal. The multi-dimensional raw data includes at least physiological signal data, behavioral activity data, and language speech data. The data preprocessing and feature engineering module is connected to the multimodal data acquisition module and is used to clean and standardize various types of raw data, and extract time-domain, frequency-domain and time-frequency-domain features related to depressive mood state to generate structured feature vectors. The multimodal feature fusion module, connected to the data preprocessing and feature engineering module, is used to receive the structured feature vector, adopt a deep learning fusion network based on the attention mechanism, dynamically learn and assign differentiated weights to different modal features, perform adaptive weighted fusion, and generate a unified deep fusion feature representation. The dynamic risk assessment model module is connected to the multimodal feature fusion module and has a built-in machine learning or deep learning model trained with labeled data. It is used to receive the deep fusion feature representation, calculate and output the user's current depression risk level, risk probability score and key influencing factor analysis. The visualization interaction and feedback module is connected to the dynamic risk assessment model module. It is used to visually present the dynamic risk assessment results and historical risk trend charts to users and authorized third parties, and to provide personalized intervention suggestions and interactive feedback interfaces.

2. The dynamic assessment system for depression risk based on multimodal data fusion according to claim 1, characterized in that, The multimodal data acquisition module specifically includes: A physiological signal acquisition unit is used to acquire at least one of heart rate variability, skin conductance, and sleep quality data through a wearable device or a built-in sensor. A behavioral activity collection unit is used to collect at least one of the following data through mobile device sensors or logs: step count, screen usage time, social application activity, and diurnal activity rhythm data. The speech acquisition unit is used to acquire speech segments through an audio input device and extract at least one of the following: speech intonation features, language content semantic features, and dialogue interaction pattern features.

3. The dynamic assessment system for depression risk based on multimodal data fusion according to claim 1, characterized in that, The multimodal feature fusion module employs a deep learning fusion network based on an attention mechanism, including: Multiple parallel modality-specific encoding subnetworks are used to process the structured feature vectors of different modalities respectively, and learn the high-level abstract representation of each modality; A cross-modal attention fusion layer is used to calculate the correlation weights between the high-level abstract representations of each modality, and to perform weighted summation or concatenation based on the weights to generate the deep fusion feature representation.

4. The dynamic assessment system for depression risk based on multimodal data fusion according to claim 1, characterized in that, The dynamic risk assessment model module includes: The short-term state assessment sub-model is used to assess a user's mood fluctuations and depression risk status at the current moment or in the short term based on the fusion features within a recent time window. The long-term trend prediction sub-model is used to predict the development trend of a user's depression risk over a future period of time based on long-term historical fusion feature sequences. The system integrates short-term condition assessment results with long-term trend prediction results to generate a final dynamic risk assessment report.

5. The dynamic assessment system for depression risk based on multimodal data fusion according to claim 1, characterized in that, The system also includes: The model continuous learning and adaptation module is connected to the dynamic risk assessment model module and the visualization interaction and feedback module. It is used to receive user feedback on the assessment results and new labeled data, and to perform online incremental updates or periodic retraining of the model parameters in the dynamic risk assessment model module to achieve personalized adaptation and continuous performance optimization of the model.

6. A dynamic assessment method for depression risk based on multimodal data fusion, characterized in that, The method, applied to the system as described in any one of claims 1-5, comprises: S1: Through the multimodal data acquisition module, continuously or periodically collect multi-dimensional raw data of the user's physiological signals, behavioral activities and speech. S2: Through the data preprocessing and feature engineering module, various types of raw data are cleaned, standardized, and feature extracted to generate structured feature vectors for each modality; S3: Through the multimodal feature fusion module, an attention-based fusion network is used to adaptively weight and fuse the structured feature vectors to generate a deep fusion feature representation; S4: Through the dynamic risk assessment model module, the deep fusion feature representation is input into the pre-trained assessment model to calculate the user's current depression risk level, risk probability, and influencing factor analysis results; S5: Through the visualization interaction and feedback module, the evaluation results are output in a visual form, and the evaluation model is optimized based on user feedback.

7. The method for dynamic assessment of depression risk based on multimodal data fusion according to claim 6, characterized in that, The adaptive weighted fusion described in step S3 specifically refers to: The attention mechanism is used to calculate the importance score of each modality feature in a specific context. Based on the importance scores, the feature representations of the corresponding modalities are dynamically weighted or selectively focused, and then fused, so that the modal features that contribute more to the current risk assessment receive higher weights.

8. The method for dynamic assessment of depression risk based on multimodal data fusion according to claim 6, characterized in that, Step S4 also includes: By combining users' historical fusion feature sequences and using time series analysis models or recurrent neural networks, dynamic change patterns of users' depression risk status can be identified, and the risk evolution trend can be reflected in the assessment results.

9. The method for dynamic assessment of depression risk based on multimodal data fusion according to claim 6, characterized in that, The method further includes: S6: Based on user confirmation and correction feedback of historical assessment results, as well as newly added labeled data, the model in the dynamic risk assessment model module is incrementally learned to continuously optimize the accuracy of personalized risk assessment for the user.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 6-9.

Images