Mental health state intelligent evaluation and early warning system based on multi-modal data fusion

Abstract

The invention relates to the technical field of psychological health monitoring, and discloses a psychological health state intelligent assessment and early warning system based on multi-modal data fusion, which comprises a multi-modal data acquisition module, a data preprocessing module, a cross-modal feature fusion module, a psychological health state assessment module, an early warning decision module and an interactive feedback module. The physiological, behavior, text and environment multi-dimensional data of the user are comprehensively collected through the multi-modal data collection module, the defect that in the prior art, single data modal evaluation is insufficient in comprehensiveness is overcome, and the psychological health state of the user can be reflected more comprehensively; meanwhile, a targeted preprocessing strategy and a timestamp alignment mechanism are adopted, so that the quality and time-space consistency of the multi-modal data are ensured, and a foundation is laid for subsequent feature fusion.

Description

Technical Field

[0001] This invention belongs to the field of mental health monitoring technology, specifically a mental health status intelligent assessment and early warning system based on multimodal data fusion. Background Technology

[0002] With the accelerating pace of society and increasing life pressures, mental health issues are receiving growing attention. Timely and accurate assessment of an individual's mental health status and effective early warning are crucial for preventing the deterioration of mental illnesses and safeguarding individual physical and mental well-being. Existing mental health assessment technologies are mainly divided into traditional assessment methods and intelligent assessment methods based on single data modalities.

[0003] Traditional assessment methods primarily rely on clinical diagnoses by psychologists and paper-based questionnaires, which suffer from low efficiency, high subjectivity, and limited coverage, making it difficult to achieve real-time monitoring and early warning for large populations. Intelligent assessment methods based on single data modalities, such as assessments using only physiological signals or text data, while improving efficiency, suffer from limited data dimensions and insufficient comprehensiveness. For example, assessing mental health solely through electrocardiogram (ECG) signals fails to reflect the impact of user psychological activities and environmental factors on mental health; assessments relying solely on text data are easily influenced by user expression habits and a tendency to conceal information, leading to low accuracy.

[0004] Existing multimodal data fusion techniques for mental health assessment mostly employ simple feature splicing or weighted fusion methods, failing to fully consider the correlation and complementarity of data from different modalities, resulting in poor fusion effects. Furthermore, early warning mechanisms often use fixed thresholds, unable to dynamically adjust based on individual user differences and historical status, leading to insufficient accuracy and targeting of warnings. In addition, existing systems lack precise localization of abnormal features and personalized intervention suggestions, failing to meet users' individual needs and limiting the effectiveness of interventions after warnings.

[0005] Therefore, in view of the problems existing in the current technology, such as insufficient comprehensiveness of assessment, poor integration effect, rigid early warning mechanism and low degree of personalization, there is an urgent need for an intelligent assessment and early warning system for mental health status that can achieve deep integration of multimodal data, dynamic adjustment of early warning threshold, accurate location of abnormal features and personalized intervention.

[0006] Based on this, an intelligent assessment and early warning system for mental health status based on multimodal data fusion was designed. Summary of the Invention

[0007] In order to overcome the shortcomings of the prior art, this invention provides an intelligent assessment and early warning system for mental health status based on multimodal data fusion, which effectively solves the problems raised in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: an intelligent assessment and early warning system for mental health status based on multimodal data fusion, comprising: a multimodal data acquisition module, a data preprocessing module, a cross-modal feature fusion module, a mental health status assessment module, an early warning decision module, and an interactive feedback module; in, The multimodal data acquisition module is used to simultaneously acquire the user's physiological signal data, behavioral data, text data, and environmental perception data. The physiological signal data includes electrocardiogram (ECG) signals, electroencephalogram (EEG) signals, skin conductance signals, and respiratory rate signals. The behavioral data includes facial expression data, limb movement data, and sleep behavior data. The text data includes the user's social text data and psychological questionnaire response text data. The environmental perception data includes the light intensity, noise level, and temperature and humidity data of the user's environment. The data preprocessing module is connected to the multimodal data acquisition module and is used to perform targeted preprocessing on the acquired modal data, including wavelet threshold noise reduction on physiological signal data, key frame extraction and posture normalization on behavioral data, word segmentation, stop word removal and word embedding on text data, outlier removal and data standardization on environmental perception data, and timestamp alignment of each modal data. The cross-modal feature fusion module is connected to the data preprocessing module. It is used to extract local and global features of each preprocessed modal data, calculate the attention weight of each modal feature using an attention mechanism, and perform weighted fusion of multimodal features using an adaptive weight allocation algorithm to output a fused feature vector. The mental health status assessment module is connected to the cross-modal feature fusion module. Based on the fused feature vector, a multi-task learning model is constructed. The multi-task learning model includes a main task layer and an auxiliary task layer. The main task layer is used to complete the mental health status level assessment and output four levels of results: normal, mildly abnormal, moderately abnormal, and severely abnormal. The auxiliary task layer is used to complete the abnormal feature localization and output the core modality and key features that cause the abnormality. The early warning decision module is connected to the mental health status assessment module and the multimodal data acquisition module, respectively. It is used to construct a dynamic early warning threshold model based on the mental health status level assessment results, abnormal feature location results and user historical data, set three early warning levels of Level 1, Level 2 and Level 3, and generate corresponding early warning information. The interactive feedback module is connected to the mental health status assessment module and the early warning decision module, respectively. It is used to display the mental health status assessment results, abnormal feature analysis reports and early warning information. At the same time, it pushes personalized intervention suggestions based on the assessment and early warning results, receives user feedback information and transmits it to the early warning decision module for threshold optimization.

[0009] Preferably, the multimodal data acquisition module includes a physiological signal acquisition unit, a behavioral data acquisition unit, a text data acquisition unit, and an environmental perception unit; The physiological signal acquisition unit uses a wearable sensor array, including an electrocardiogram sensor, an electroencephalogram sensor, a skin conductance sensor, and a respiration sensor, with a sampling frequency of not less than 256Hz; The behavior data acquisition unit includes a high-definition camera and a motion capture sensor. The high-definition camera has a frame rate of no less than 30fps, and the motion capture sensor uses an inertial measurement unit. The text data acquisition unit obtains social text data through the user terminal API interface and collects response text data through the built-in questionnaire module. The environmental sensing unit employs an environmental sensor array, including a light sensor, a noise sensor, and a temperature and humidity sensor.

[0010] Preferably, the timestamp alignment of the data preprocessing module specifically involves: using the timestamp of physiological signal acquisition as a benchmark, calibrating the timestamps of behavioral data, text data, and environmental perception data through a linear interpolation algorithm to unify the temporal resolution of each modality data to 1 s / frame, ensuring the corresponding matching of multimodal data in the same time dimension.

[0011] Preferably, the specific working process of the cross-modal feature fusion module is as follows: S1: Convolutional neural networks are used to extract local features of physiological signal data and behavioral data, Transformer models are used to extract local features and global semantic features of text data, and gradient boosting trees are used to extract global statistical features of environmental perception data. S2: Construct a cross-modal attention mechanism model, calculate the correlation score between each modal feature and the mental health status assessment task, and use the correlation score as the attention weight; S3: Based on the signal-to-noise ratio and data integrity of each modality, the attention weights are corrected through an adaptive weight allocation algorithm to obtain the final fusion weights; S4: The features of each modality are weighted and summed according to the fusion weights, and the features are standardized through the BatchNorm layer to output a fusion feature vector with uniform dimensions.

[0012] Preferably, the multi-task learning model of the mental health status assessment module adopts a shared encoder-independent decoder structure. The shared encoder uses a deep residual network to perform deep encoding on the fused feature vector. The main task decoder uses a fully connected layer combined with a Softmax function to output the probability distribution of mental health status levels. The auxiliary task decoder uses a convolutional attention module to locate abnormal core modalities and key features. The main task and auxiliary task are trained by a joint loss function, which is a weighted sum of cross-entropy loss and mean squared error loss.

[0013] Preferably, the dynamic early warning threshold model of the early warning decision module is as follows: a time series prediction model is constructed based on the user's historical mental health status assessment results to predict the trend of the user's mental health status change in the future period. Combined with the benchmark data of the group's mental health status, the early warning threshold is dynamically adjusted through the Bayesian inference algorithm. Different early warning levels correspond to different intervention response mechanisms: a level 1 early warning triggers the system to automatically push psychological counseling articles, a level 2 early warning triggers human customer service intervention and communication, and a level 3 early warning triggers emergency contact notification and connection with professional psychologists.

[0014] Preferably, it also includes a data security encryption module, which is connected to the multimodal data acquisition module and the interactive feedback module respectively. It is used to perform end-to-end encrypted storage of the acquired user multimodal data, encrypt the transmitted evaluation results and early warning information using the AES encryption algorithm, and implement access control for user data, allowing only authorized users and professional physicians to view the corresponding data.

[0015] Preferably, the interactive feedback module includes a display unit, an input unit, and a push unit; The display unit uses a touch screen to display assessment reports, early warning information, and intervention suggestions; The input unit is used to receive user feedback on the evaluation results and the implementation status of intervention suggestions; The push unit sends early warning information and intervention suggestions through three methods: SMS, APP push, and email.

[0016] Preferably, the cross-modal feature fusion module is further provided with a feature verification unit, which is used to verify the effectiveness of the fused features by using variance analysis and correlation analysis, eliminate redundant features, and improve the recognizability of the fused features.

[0017] Preferably, the early warning decision module further includes an early warning effect evaluation unit, which is used to evaluate the rationality of the early warning level and the effectiveness of the intervention suggestions based on the user's subsequent changes in mental health status and user feedback information, and feed the evaluation results back to the cross-modal feature fusion module and the mental health status evaluation module to achieve dynamic optimization of model parameters.

[0018] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention comprehensively collects users' physiological, behavioral, textual, and environmental data through a multimodal data acquisition module, making up for the shortcomings of existing technologies in terms of comprehensiveness of single-data modality assessment, and can more comprehensively reflect users' mental health status; at the same time, it adopts targeted preprocessing strategies and timestamp alignment mechanisms to ensure the quality and spatiotemporal consistency of multimodal data, laying the foundation for subsequent feature fusion. 2. The cross-modal feature fusion module of the present invention adopts a fusion strategy that combines attention mechanism and adaptive weight allocation algorithm. It not only considers the correlation between each modal feature and the evaluation task, but also dynamically adjusts the fusion weights based on the quality of the data itself (signal-to-noise ratio, integrity), thereby realizing deep fusion of multimodal features. Compared with the existing simple splicing or fixed weight fusion methods, it significantly improves the effectiveness and recognizability of fused features, thereby improving the evaluation accuracy. 3. This invention constructs a multi-task learning model that can accurately locate abnormal features while completing the assessment of mental health status levels. It can clearly identify the core modalities and key features that lead to users' psychological abnormalities, providing a precise basis for subsequent personalized intervention. The multi-task joint training method improves the model's generalization ability and assessment stability. 4. The early warning decision module of this invention adopts a dynamic early warning threshold model, which combines individual user historical data and group benchmark data, and dynamically adjusts the early warning threshold through Bayesian inference. Compared with the existing fixed threshold early warning mechanism, it is more personalized and adaptable. At the same time, it sets up a graded early warning and corresponding intervention response mechanism, realizing full-process coverage from mild guidance to emergency intervention, and improving the practicality and effectiveness of early warning. 5. This invention adds a data security encryption module to achieve end-to-end encrypted storage and transmission of user data, ensuring user privacy and security; the interactive feedback module supports multi-channel information push and user feedback collection, forming a closed-loop mechanism of evaluation-early warning-intervention-optimization, continuously improving system performance. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0020] In the attached diagram: Figure 1 This is a block diagram of the overall structure of the system of the present invention; Figure 2 This is a flowchart illustrating the workflow of the cross-modal feature fusion module of the present invention. Figure 3 This is the overall flowchart of the mental health status assessment and early warning system of this invention; In the diagram: 1. Multimodal data acquisition module; 11. Physiological signal acquisition unit; 12. Behavioral data acquisition unit; 13. Text data acquisition unit; 14. Environmental perception unit; 2. Data preprocessing module; 3. Cross-modal feature fusion module; 31. Feature extraction unit; 32. Attention weight calculation unit; 33. Adaptive weight correction unit; 34. Feature fusion unit; 35. Feature verification unit; 4. Mental health status assessment module; 41. Shared encoder; 42. Main task decoder; 43. Auxiliary task decoder; 5. Early warning decision module; 51. Dynamic threshold construction unit; 52. Early warning level determination unit; 53. Early warning effect evaluation unit; 6. Interactive feedback module; 61. Display unit; 62. Input unit; 63. Push unit; 7. Data security encryption module. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, this invention provides an intelligent assessment and early warning system for mental health status based on multimodal data fusion, including: a multimodal data acquisition module 1, a data preprocessing module 2, a cross-modal feature fusion module 3, a mental health status assessment module 4, an early warning decision module 5, an interactive feedback module 6, and a data security encryption module 7; the modules transmit data through a high-speed data bus to ensure the real-time performance and stability of data transmission.

[0023] The multimodal data acquisition module 1 is used to synchronously acquire multi-dimensional data of users, including a physiological signal acquisition unit 11, a behavior data acquisition unit 12, a text data acquisition unit 13, and an environmental perception unit 14; the physiological signal acquisition unit 11 adopts an integrated design of a wearable bracelet and a headband, with an electrocardiogram sensor, an electroencephalogram sensor, a galvanic skin sensor, and a respiratory sensor built-in, and the sampling frequency is set to 512Hz, which can accurately acquire key signals such as the electrocardiogram R wave, electroencephalogram alpha wave, and beta wave of users; the behavior data acquisition unit 12 includes high-definition cameras (frame rate 60fps) deployed in the user's living environment and inertial measurement units worn on the user's wrists and waists. The high-definition cameras are used to acquire facial expression images, and the inertial measurement units are used to acquire limb movement data. At the same time, sleep behavior data such as the user's sleep duration and sleep depth are acquired through wearable devices; the text data acquisition unit 13 obtains text data from social platforms such as WeChat and Weibo through the user's mobile phone APP authorization interface, and collects user response text data through a standardized psychological questionnaire (such as the SCL-90 scale) built into the system; the environmental perception unit 14 adopts an environmental sensor array deployed in the user's activity area to collect light intensity (unit: lux), noise decibels (unit: dB), and temperature and humidity (unit: °C, %RH) data in real time.

[0024] The data preprocessing module 2 receives the data of each modality transmitted by the multimodal data acquisition module 1 and performs targeted preprocessing; for physiological signal data, wavelet threshold denoising processing is adopted, the db4 wavelet basis is selected, and the decomposition layer is 5 layers. The power frequency interference and electromyogram interference in the electrocardiogram and electroencephalogram signals are removed by calculating the threshold; for behavior data, the frame difference method is used to extract the key frames of facial expressions and limb movements, and the pose normalization is achieved through affine transformation to unify the size and angle of the key frames; for text data, the jieba word segmentation tool is used for word segmentation, stop words such as "de" and "le" are removed, and the text words are converted into 300-dimensional word embedding vectors through the Word2Vec model; for environmental perception data, the 3σ criterion is used to剔除异常值 (remove outliers), and the data is mapped to the [0,1] interval through min-max normalization; at the same time, based on the time stamp of physiological signal acquisition, the time stamps of behavior data, text data, and environmental perception data are calibrated through a linear interpolation algorithm, so that the time resolution of each modality data is unified to 1s / frame, ensuring the corresponding matching of multimodal data in the same time dimension.

[0025] Such as Figure 2 It should be noted that the phrase "剔除异常值" in the original text is replaced with "remove outliers" in the translation, and there may be a more accurate term in English for this specific concept in the context of data processing. You can adjust it according to the actual situation.As shown, the cross-modal feature fusion module 3 includes a feature extraction unit 31, an attention weight calculation unit 32, an adaptive weight correction unit 33, a feature fusion unit 34, and a feature verification unit 35. The feature extraction unit 31 uses a convolutional neural network (CNN) to extract local features from physiological signal data and behavioral data. Specifically, it employs a CNN structure with 4 convolutional layers and 2 pooling layers, outputting a 512-dimensional local feature vector. It also uses a Transformer model to extract local and global semantic features from text data. The Transformer model has 6 encoder layers and outputs a 512-dimensional semantic feature vector. Finally, it uses a gradient boosting tree (XG) model to extract local and global semantic features from text data. Boost extracts global statistical features from environmental perception data, outputting a 128-dimensional statistical feature vector. Attention weight calculation unit 32 constructs a cross-modal attention mechanism model, calculating the correlation score between each modality's features and the mental health status assessment task. Specifically, it calculates the cosine similarity between each modality's feature vector and the assessment task's label vector, normalizing the similarity score and using it as the attention weight. Adaptive weight correction unit 33 corrects the attention weight based on the signal-to-noise ratio and data integrity of each modality's data. For data modalities with high signal-to-noise ratio and good integrity, its weight is increased; conversely, its weight is decreased. The final fusion weight is obtained after correction. Feature fusion unit 34 performs a weighted summation of each modality's feature vector according to the fusion weight, standardizes the features through a BatchNorm layer, and outputs a 1024-dimensional fusion feature vector. Feature verification unit 35 uses analysis of variance (ANOVA) and correlation analysis to verify the effectiveness of the fusion features, eliminating redundant features with small variance and low correlation with the assessment results, further improving the discriminability of the fusion features.

[0026] The mental health status assessment module 4 constructs a multi-task learning model based on fused feature vectors. This model adopts a shared encoder-independent decoder structure. The shared encoder 41 uses a deep residual network (ResNet-50) to deeply encode the fused feature vectors. The gradient vanishing problem of deep networks is solved through residual connections, and a 2048-dimensional deep feature vector is output. The main task decoder 42 uses three fully connected layers combined with the Softmax function to output the probability distribution of four levels: normal, mildly abnormal, moderately abnormal, and severely abnormal. The level with the highest probability is selected as the mental health status assessment result. The auxiliary task decoder 43 uses a convolutional attention module (CBAM) to locate the core abnormal modalities and key features. Through channel attention and spatial attention calculations, the modality with the greatest impact on the assessment result and the key features under this modality (such as the RR interval of the electrocardiogram signal, negative emotion words in the text data, etc.) are determined. The main task and auxiliary task are trained by a joint loss function. The joint loss function is a weighted sum of cross-entropy loss (used for the main task level assessment) and mean squared error loss (used for the auxiliary task feature localization). The weight coefficients are determined to be 0.7 and 0.3 through cross-validation.

[0027] The early warning decision module 5 includes a dynamic threshold construction unit 51, an early warning level determination unit 52, and an early warning effect evaluation unit 53. The dynamic threshold construction unit 51 constructs a time series prediction model (ARIMA model) based on the user's historical mental health status assessment results to predict the user's mental health status trend over the next 7 days. Simultaneously, it collects multimodal data from a large number of healthy individuals to construct a group mental health status benchmark. Using a Bayesian inference algorithm, it combines individual prediction trends with the group benchmark data to dynamically adjust the early warning threshold. The early warning level determination unit 52 sets three early warning levels: Level 1, Level 2, and Level 3. A Level 1 early warning is triggered when the assessment result is slightly abnormal and the prediction trend is stable; a Level 3 early warning is triggered when the assessment result is slightly abnormal and the prediction trend worsens, or when the assessment result is moderately abnormal. When the assessment result is moderately abnormal and the predicted trend is worsening, or when the assessment result is severely abnormal, a level 3 warning is triggered. Different warning levels correspond to different intervention response mechanisms: Level 1 warnings trigger the system to automatically push psychological counseling articles, meditation audio, etc.; Level 2 warnings trigger human customer service intervention, communicating one-on-one with the user through the interactive feedback module; Level 3 warnings trigger emergency contact notification (via SMS or telephone) and connection with professional psychologists to provide emergency intervention services; The warning effect evaluation unit 53 evaluates the rationality of the warning level and the effectiveness of the intervention suggestions based on the user's subsequent changes in mental health status and user feedback information, and feeds the evaluation results back to the cross-modal feature fusion module 3 and the mental health status evaluation module 4 to achieve dynamic optimization of model parameters.

[0028] The interactive feedback module 6 includes a display unit 61, an input unit 62, and a push unit 63. The display unit 61 uses a 10.1-inch touch screen to display the mental health status assessment report (including grade results and abnormal feature analysis), early warning information, and personalized intervention suggestions. The input unit 62 supports touch input and voice input to receive user feedback on the assessment results (such as whether they agree with the assessment results) and the implementation status of intervention suggestions (such as whether they have completed the counseling exercises). The push unit 63 pushes early warning information and intervention suggestions through SMS, APP push, and email to ensure that users can receive relevant information in a timely manner.

[0029] The data security encryption module 7 is connected to the multimodal data acquisition module 1 and the interactive feedback module 6 respectively. It performs end-to-end encrypted storage of the collected user multimodal data and uses the AES-256 encryption algorithm to encrypt the transmitted evaluation results and warning information. At the same time, a user data access control mechanism is established, which only authorizes the user and professional psychologists (with user authorization) to view the corresponding data, effectively protecting user privacy and security.

[0030] like Figure 3 As shown, the overall process of mental health status assessment and early warning in this invention is as follows: Step 1: Multimodal data acquisition module 1 synchronously acquires the user's physiological signals, behavior, text, and environmental perception data; Step 2: Data preprocessing module 2 performs targeted preprocessing and timestamp alignment on the data for each modality; Step 3: The cross-modal feature fusion module 3 extracts features from each modality, and achieves deep fusion of multimodal features through attention mechanism and adaptive weight allocation, outputting a fused feature vector; Step 4: The mental health status assessment module 4 completes the mental health status level assessment and abnormal feature localization based on the fusion feature vector through a multi-task learning model; Step 5: The early warning decision module 5 combines the evaluation results, abnormal characteristics, and user historical data to determine the early warning level through a dynamic early warning threshold model and generate early warning information; Step 6: The interactive feedback module 6 displays the assessment results, early warning information, and personalized intervention suggestions, and receives user feedback; Step 7: The early warning effect evaluation unit 53 evaluates the early warning effect based on the user's subsequent status changes and feedback information, and provides feedback to optimize the model parameters; Step 8: Repeat steps 1-7 to achieve continuous monitoring, assessment and early warning of the user's mental health status.

[0031] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0032] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A mental health status intelligent assessment and early warning system based on multimodal data fusion, characterized in that, include: The system includes a multimodal data acquisition module, a data preprocessing module, a cross-modal feature fusion module, a mental health status assessment module, an early warning decision-making module, and an interactive feedback module. in, The multimodal data acquisition module is used to simultaneously acquire the user's physiological signal data, behavioral data, text data, and environmental perception data. The physiological signal data includes electrocardiogram (ECG) signals, electroencephalogram (EEG) signals, skin conductance signals, and respiratory rate signals. The behavioral data includes facial expression data, limb movement data, and sleep behavior data. The text data includes the user's social text data and psychological questionnaire response text data. The environmental perception data includes the light intensity, noise level, and temperature and humidity data of the user's environment. The data preprocessing module is connected to the multimodal data acquisition module and is used to perform targeted preprocessing on the acquired modal data, including wavelet threshold noise reduction on physiological signal data, key frame extraction and posture normalization on behavioral data, word segmentation, stop word removal and word embedding on text data, outlier removal and data standardization on environmental perception data, and timestamp alignment of each modal data. The cross-modal feature fusion module is connected to the data preprocessing module. It is used to extract local and global features of each preprocessed modal data, calculate the attention weight of each modal feature using an attention mechanism, and perform weighted fusion of multimodal features using an adaptive weight allocation algorithm to output a fused feature vector. The mental health status assessment module is connected to the cross-modal feature fusion module. Based on the fused feature vector, a multi-task learning model is constructed. The multi-task learning model includes a main task layer and an auxiliary task layer. The main task layer is used to complete the mental health status level assessment and output four levels of results: normal, mildly abnormal, moderately abnormal, and severely abnormal. The auxiliary task layer is used to complete the abnormal feature localization and output the core modality and key features that cause the abnormality. The early warning decision module is connected to the mental health status assessment module and the multimodal data acquisition module, respectively. It is used to construct a dynamic early warning threshold model based on the mental health status level assessment results, abnormal feature location results and user historical data, set three early warning levels of Level 1, Level 2 and Level 3, and generate corresponding early warning information. The interactive feedback module is connected to the mental health status assessment module and the early warning decision module, respectively. It is used to display the mental health status assessment results, abnormal feature analysis reports and early warning information. At the same time, it pushes personalized intervention suggestions based on the assessment and early warning results, receives user feedback information and transmits it to the early warning decision module for threshold optimization.

2. The intelligent assessment and early warning system for mental health status based on multimodal data fusion according to claim 1, characterized in that: The multimodal data acquisition module includes a physiological signal acquisition unit, a behavioral data acquisition unit, a text data acquisition unit, and an environmental perception unit. The physiological signal acquisition unit uses a wearable sensor array, including an electrocardiogram sensor, an electroencephalogram sensor, a skin conductance sensor, and a respiration sensor, with a sampling frequency of not less than 256Hz; The behavior data acquisition unit includes a high-definition camera and a motion capture sensor. The high-definition camera has a frame rate of no less than 30fps, and the motion capture sensor uses an inertial measurement unit. The text data acquisition unit obtains social text data through the user terminal API interface and collects response text data through the built-in questionnaire module. The environmental sensing unit employs an environmental sensor array, including a light sensor, a noise sensor, and a temperature and humidity sensor.

3. The intelligent assessment and early warning system for mental health status based on multimodal data fusion according to claim 1, characterized in that: The timestamp alignment of the data preprocessing module is specifically as follows: using the timestamp of physiological signal acquisition as a benchmark, the timestamps of behavioral data, text data and environmental perception data are calibrated through a linear interpolation algorithm to unify the time resolution of each modality data to 1s / frame, ensuring the corresponding matching of multimodal data in the same time dimension.

4. The intelligent assessment and early warning system for mental health status based on multimodal data fusion according to claim 1, characterized in that: The specific working process of the cross-modal feature fusion module is as follows: S1: Convolutional neural networks are used to extract local features of physiological signal data and behavioral data, Transformer models are used to extract local features and global semantic features of text data, and gradient boosting trees are used to extract global statistical features of environmental perception data. S2: Construct a cross-modal attention mechanism model, calculate the correlation score between each modal feature and the mental health status assessment task, and use the correlation score as the attention weight; S3: Based on the signal-to-noise ratio and data integrity of each modality, the attention weights are corrected through an adaptive weight allocation algorithm to obtain the final fusion weights; S4: The features of each modality are weighted and summed according to the fusion weights, and the features are standardized through the BatchNorm layer to output a fusion feature vector with uniform dimensions.

5. The intelligent assessment and early warning system for mental health status based on multimodal data fusion according to claim 1, characterized in that: The multi-task learning model of the mental health status assessment module adopts a shared encoder-independent decoder structure. The shared encoder uses a deep residual network to deeply encode the fused feature vector. The main task decoder uses a fully connected layer combined with a Softmax function to output the probability distribution of mental health status levels. The auxiliary task decoder uses a convolutional attention module to locate abnormal core modalities and key features. The main task and auxiliary task are trained by a joint loss function, which is a weighted sum of cross-entropy loss and mean squared error loss.

6. The intelligent assessment and early warning system for mental health status based on multimodal data fusion according to claim 1, characterized in that: The dynamic early warning threshold model of the early warning decision module is as follows: Based on the user's historical mental health status assessment results, a time series prediction model is constructed to predict the trend of the user's mental health status change in the future. Combined with the benchmark data of the group's mental health status, the early warning threshold is dynamically adjusted through the Bayesian inference algorithm. Different early warning levels correspond to different intervention response mechanisms: Level 1 early warning triggers the system to automatically push psychological counseling articles, Level 2 early warning triggers human customer service intervention and communication, and Level 3 early warning triggers emergency contact notification and connection with professional psychologists.

7. The intelligent assessment and early warning system for mental health status based on multimodal data fusion according to claim 1, characterized in that: It also includes a data security encryption module, which is connected to the multimodal data acquisition module and the interactive feedback module respectively. It is used to perform end-to-end encrypted storage of the collected user multimodal data, encrypt the transmitted evaluation results and early warning information using the AES encryption algorithm, and implement access control for user data, allowing only authorized users and professional physicians to view the corresponding data.

8. The intelligent assessment and early warning system for mental health status based on multimodal data fusion according to claim 1, characterized in that: The interactive feedback module includes a display unit, an input unit, and a push unit; The display unit uses a touch screen to display assessment reports, early warning information, and intervention suggestions; The input unit is used to receive user feedback on the evaluation results and the implementation status of intervention suggestions; The push unit sends early warning information and intervention suggestions through three methods: SMS, APP push, and email.

9. The intelligent assessment and early warning system for mental health status based on multimodal data fusion according to claim 1, characterized in that: The cross-modal feature fusion module is also equipped with a feature verification unit, which is used to verify the effectiveness of the fused features using variance analysis and correlation analysis, eliminate redundant features, and improve the recognizability of the fused features.

10. The intelligent assessment and early warning system for mental health status based on multimodal data fusion according to claim 1, characterized in that: The early warning decision module also includes an early warning effect evaluation unit, which evaluates the rationality of the early warning level and the effectiveness of the intervention suggestions based on the user's subsequent changes in mental health status and user feedback information. The evaluation results are fed back to the cross-modal feature fusion module and the mental health status evaluation module to achieve dynamic optimization of model parameters.

Images