Mental health state real-time early warning method and device based on multi-modal data fusion

Abstract

The invention discloses a psychological health state real-time early warning method and device based on multi-modal data fusion. According to the method, the fusion precision of the multi-modal data is remarkably improved through a double-circulation fusion mechanism. The inner ring attention mechanism can dynamically adjust the weight according to the real-time features of different modal data, and it is ensured that key information can be preferentially captured when abnormal fluctuation occurs to physiological signals or text emotion changes suddenly; the adaptability of a fusion strategy is further enhanced by an environment context variable introduced by an outer ring, so that the system can still keep stable feature extraction capability under external conditions such as illumination in different seasons, a finally generated comprehensive feature vector is closer to a real psychological health state, the input quality of an early warning model is fundamentally improved, and the early warning efficiency is improved. A real-time monitoring and dynamic updating mechanism effectively enhances the timeliness and personalized adaptation capability of the early warning system, the false alarm rate is remarkably reduced, and an early warning result better conforms to the actual situation of a user.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of mental health early warning, and particularly relates to a mental health state real-time early warning method based on multi-modal data fusion and a device thereof. BACKGROUND

[0002] Mental health state early warning refers to a systematic process of early identification, dynamic assessment and risk prompt of individual mental health status through multi-dimensional psychological index monitoring and data analysis. The mechanism relies on psychological assessment scales, behavior observation data, physiological signal monitoring and artificial intelligence algorithms to build a comprehensive evaluation model, real-time capture key signals such as emotional fluctuation, stress level and cognitive function change, and identify potential psychological risks such as anxiety, depression or stress response. The early warning system can generate individualized risk level reports in combination with historical data and group benchmarks, and push intervention suggestions to users or professional institutions through an intelligent platform, realizing early detection, early intervention and early relief. Mental health state early warning not only provides self-adjustment basis for individuals, but also provides scientific support for medical institutions, educational units and enterprise organizations to carry out precise mental health services and crisis intervention, and helps to build a prevention-oriented and whole-process management mental health security system.

[0003] However, the prior art adopts static weight distribution when fusing multi-modal data, which cannot dynamically adjust the importance of each modality according to real-time data characteristics and environmental variables, resulting in delayed capture of key information; the early warning model has a long update cycle and lacks a personalized adaptation mechanism, making it difficult to quickly respond to changes in user behavior patterns, resulting in a decline in early warning effectiveness over time. SUMMARY

[0004] The purpose of the present application is to solve the above-mentioned problems, and to provide a mental health state real-time early warning method based on multi-modal data fusion and a device thereof.

[0005] The technical solution adopted by the present application is as follows: a mental health state real-time early warning method based on multi-modal data fusion, the method comprising the following steps: S1: Collect physiological signals such as heart rate and electrodermal activity through wearable devices, synchronously acquire behavior data such as user social media dynamics, voice recordings and mobile terminal usage time, establish a real-time transmission channel for multi-source data, and ensure accurate alignment of various data on the time axis, laying a foundation for subsequent fusion analysis.

[0006] S2: Clean and denoise the collected raw data, use an adaptive threshold method to remove motion artifacts in physiological signals, and use natural language processing technology to perform word segmentation and sentiment annotation on text data. The standardized data set output in this step will be directly used in the feature extraction stage.

[0007] S3: Based on the pre-processed data, a multi-dimensional feature space is constructed, time-domain features such as heart rate variability are extracted from physiological signals, and semantic features such as sentiment polarity are mined from text data. A transfer learning method is used to optimize the feature extraction model to adapt to different user data distributions. The feature extraction results will be used as the input layer of the fusion network.

[0008] S4: A double-cycle fusion mechanism is designed. The inner ring dynamically allocates the feature weights of each modality through attention mechanism, and the outer ring introduces environmental context variables such as season and light to adjust the fusion strategy. The multi-dimensional features obtained in the feature extraction stage are input into the network to generate a comprehensive feature vector with context adaptability.

[0009] S5: A semi-supervised learning method is used to build a two-layer model containing an anomaly detection layer and a risk classification layer. The comprehensive features output by the fusion network are used for training. The anomaly detection layer is responsible for identifying data patterns deviating from the baseline, and the risk classification layer outputs warning levels combined with the historical case library. During model training, feedback data from the effect evaluation stage will be continuously received.

[0010] S6: Deploy the trained warning model to edge computing devices and perform real-time inference on new input fusion features. Every 24 hours, the incremental learning module is started to update the model parameters using the new data from the previous day, ensuring that the detection accuracy is dynamically improved over time.

[0011] S7: When the model detects that the risk level exceeds the preset threshold, a multi-dimensional warning mechanism is started, personalized psychological adjustment suggestions are pushed to the user, and the warning information is encrypted and transmitted to the designated guardian terminal. The warning rules will be dynamically adjusted according to the intervention effect in the user feedback stage.

[0012] S8: Establish an interactive feedback interface to collect user evaluations of warning results and the implementation of intervention measures. Reinforcement learning is used to convert user feedback into model optimization signals, directly affecting the weight distribution strategy of the fusion network, forming a closed-loop regulation of "warning-intervention-feedback".

[0013] S9: Cross-cycle effect evaluation. Quarterly longitudinal evaluation is carried out to compare the changes in mental health indicators before and after intervention, analyze the warning contribution of different modal data, and feed back the evaluation results to the data collection stage to optimize the sensor configuration scheme, and provide improvement basis for the context adjustment mechanism of the fusion network.

[0014] In a preferred embodiment, in the step S1, the time axis accurate alignment in the multi-modal data collaborative collection is realized through a three-stage synchronization mechanism. First, the GPS timing module is combined with the local clock calibration to configure the Beidou satellite timestamp for the wearable device, ensuring that the physiological signal sampling interval error is controlled within ±5 ms. The social media data is automatically attached with the UTC time generated by the server when obtained through the API interface, and the mobile terminal uses the time when the screen is turned on and off to the second level through the system level log. In the data transmission stage, the edge node caching strategy is adopted, and when the network delay exceeds 200 ms, local storage is triggered, and after the network is restored, breakpoint resume is performed through the timestamp comparison tool. All data needs to pass through the time axis verification algorithm before entering the preprocessing stage, and if the timestamp deviation of a certain modal data exceeds the threshold, the resampling mechanism is started to ensure that the physiological, text, and behavior data form a complete sample in the same time window.

[0015] In a preferred embodiment, in the step S2, the motion artifact rejection adopts a processing flow combining adaptive threshold method and wavelet transform. First, the original physiological signal is decomposed by 5 layers of wavelet, the high-frequency noise component is extracted, and the energy spectral density is calculated. The frequency band with energy proportion less than 0.01 is defined as the artifact interval. Then a sliding window is constructed, and the window size is dynamically adjusted according to the signal sampling rate. The standard deviation and kurtosis value of the data in the window are calculated, and when the kurtosis value exceeds 3.5, the threshold sensitivity is automatically increased. The time sequence characteristics of the original signal are preserved during processing, and the missing values after artifact rejection are filled by interpolation method to ensure that the length of the preprocessed data is consistent with the original signal. The standardized data set is packaged in JSON-LD format, and each data sample contains three tuples of modal identification, timestamp, and feature value, which is directly mapped to the input tensor dimension of the feature extraction model.

[0016] In a preferred embodiment, in the step S3, the transfer learning optimized feature extraction model is realized through a two-stage training. In the first stage, the base model is pre-trained on the public data set. The physiological signal feature extraction adopts the CNN-LSTM hybrid architecture, the text feature extraction uses the second last layer output of the BERT-base model, and the behavior feature is captured through the graph neural network to capture the user interaction topological structure. In the second stage, user individual calibration is introduced, the stable period data of the previous two weeks is collected to construct a private sample set, and the knowledge distillation technology is used to transfer the feature extraction ability of the base model to the lightweight model, while the bottom weights are frozen and only the top fully connected layer is fine-tuned. During model optimization, the JS divergence of feature space distribution is monitored, and when the divergence value is less than 0.15, the transfer is stopped to ensure that the model can adapt to the new user data distribution and retain the general feature extraction ability. The extracted multi-dimensional features are mapped to the [-1, 1] interval through the standardization layer, and directly used as the tensor data of the input layer of the fusion network.

[0017] In a preferred embodiment, the step S4, the inner loop attention weight distribution cooperates with the outer loop environmental adjustment, first taking the physiological signal feature set X = {x1, x2,..., x m} (such as heart rate variability, electrodermal activity), the text sentiment feature set Y = {y1, y2,..., y n} (such as sentiment polarity, semantic intensity) and the behavior feature set Z = {z1, z2,..., z k} (such as social media interaction frequency, screen use time length) as input.

[0018] The inner loop calculates the initial attention weight of each modality through a multi-layer perception (MLP), wherein the importance score Sx of the physiological signal modality is determined by its deviation from the historical health baseline, the text modality score Sy is based on the product of sentiment word frequency and negative word modification intensity, and the behavior modality score Sz combines abnormal fluctuation of use time length and social isolation index. The above scores are input into a softmax function to obtain the inner loop basic weight α = [α x , α y , α z ], the weight value is dynamically updated according to the real-time data distribution, for example, when strong negative sentiment is detected in the text data, α y is automatically increased by 20% to 30%. The outer loop introduces environmental context variables C = {c1, c2,..., c t} (such as seasonal coefficient c season , light intensity c light , time period c time ), and modifies the inner loop weight through the design of environmental adjustment factor β. Specifically, first, an environmental impact matrix M ∈ Rtx3 is constructed, wherein element m i,j represents the influence coefficient of the i-th environmental variable on the j-th modality (for example, when c season =0.8, the physiological signal modality influence coefficient m1,1=1.2, which enhances its weight proportion), then the environmental factors are converted into a weight adjustment vector through β = sigmoid(M·α), and finally the final fusion weight w is calculated by the Hadamard product of the inner loop basic weight and the outer loop adjustment vector, that is, w = α ⊙ β. Based on this weight, the multi-modal features are linearly fused to generate a comprehensive feature vector V = w x X + w y Y + w z Z, wherein w x , w y , w z are the corrected physiological, text, and behavior modality weights, respectively. The environmental perception attention weight correction formula is: ; wj is the final fusion weight for the j-th modality; S j is the initial importance score of the j-th modality; m i,j is the influence coefficient of the i-th environmental variable on the j-th modality in the environmental influence matrix; c i is the normalized value of the i-th environmental context variable (e.g. converting light intensity to a 0-1 interval value); t is the total number of environmental variables; k is the total number of modalities (here k = 3, corresponding to physiological, textual, and behavioral modalities).

[0019] In a preferred embodiment, in step S5, the semi-supervised learning constructed double-layer model adopts a co-training framework. The anomaly detection layer uses the Isolation Forest algorithm to map the fusion feature vector to a hypersphere in a high-dimensional space, and judges the degree of deviation by calculating the distance from the sample point to the center of the sphere. The distance threshold is dynamically set by the 3σ rule combined with user historical data. The risk grading layer is based on an improved K-Nearest Neighbor algorithm, which performs cosine similarity matching between the comprehensive feature vector and the risk samples in the historical case library, and selects the weighted average of the risk levels of the Top-5 similar cases as the output. During model training, 5-fold cross-validation is used, and the model parameters with the highest AUC value on the validation set after each training are retained. At the same time, the intervention effectiveness data fed back during the effect evaluation stage is received, and when the early warning accuracy of a certain type of risk is less than 85%, the retraining process of the samples of that type is automatically triggered.

[0020] In a preferred embodiment, in step S6, the incremental learning module updates parameters through a sliding time window mechanism. The model update process is started at 3 am every day, and the newly added data in the past 24 hours is extracted from the local database of the edge device, and divided into an incremental training set and a validation set in a ratio of 8:2. Elastic weight consolidation (EWC) is used to protect key parameters, and different regularization coefficients are set for the attention layer and the environmental adjustment layer of the fusion network. The regularization strength of the attention weight is 1.5 times that of the environmental adjustment factor. During the update process, the change of F1-score on the validation set is monitored, and if the F1-score decreases by more than 5% after three consecutive updates, the historical optimal parameters are rolled back and the learning rate is adjusted. Model deployment uses TensorRT quantization acceleration, and the inference delay is controlled within 200ms, meeting the response requirements of real-time monitoring.

[0021] In a preferred embodiment, in step S7, personalized psychological adjustment suggestions are generated using a hybrid strategy based on a rule engine and case-based reasoning. The rule engine incorporates a three-tiered intervention plan: for mild risks, breathing training audio is pushed (duration set to 5 or 10 minutes based on the user's historical preferences); for moderate risks, short cognitive behavioral therapy articles are added (vocabulary difficulty controlled at CEFR B1 level); and for severe risks, real-time voice guidance is triggered. The time parameters in the suggestions are synchronized with the user's circadian rhythm, for example, pushed 30 minutes in advance based on historical data analysis of emotional low points. Warning information is encrypted and transmitted using the national cryptographic SM4 algorithm. After receiving the information, the guardian's terminal needs to verify it again using a dynamic password to ensure data privacy. The adjustment cycle of the warning rules is synchronized with the user feedback phase; the threshold coefficient for each risk level is recalculated every 20 valid feedback responses collected.

[0022] In a preferred embodiment, step S8, where reinforcement learning transforms user feedback into model optimization signals, consists of four steps. First, the feedback evaluation is quantified into reward values: +1 point for "accurate warning and effective intervention," -0.5 points for "false alarm," and -0.3 points for "intervention measures not implemented." Then, a state space is constructed, containing the current fusion weight vector, environmental variable values, and the user's recent emotional trend. The value function of each action is calculated using the Q-learning algorithm, and the action space is defined as the weight adjustment amplitude (a continuous value between -0.2 and +0.2). A policy update is performed every 5 feedbacks received, directly applying the weight adjustment amount corresponding to the action with the largest Q value to the inner loop attention layer of the fusion network. Simultaneously, the impact of this adjustment on the subsequent warning accuracy is recorded, forming a decay coefficient matrix of the feedback signal.

[0023] In a preferred embodiment, in step S9, the contribution analysis of different modal data for early warning employs a combination of SHAP values ​​and a confusion matrix. First, the average absolute SHAP value of each modal feature in the early warning model is calculated, with the contribution weights for physiological signals, text, and behavioral modalities denoted as Wp, Wt, and Wb, respectively. Then, a multimodal ablation experiment is constructed, and after sequentially removing a certain modality, the decrease in model early warning accuracy Δp, Δt, and Δb is calculated. The final contribution score is obtained through a weighted summation: Score = 0.6 × SHAP value + 0.4 × accuracy decrease. In the evaluation results, if the Score of a certain modality is below 0.3 for two consecutive quarters, feedback is sent to the data acquisition stage to adjust the sampling frequency of the corresponding sensor, for example, reducing the acquisition interval of behavioral data or increasing the number of sensors for physiological signals. Simultaneously, the influence coefficient of environmental variables is incorporated into the contribution analysis, providing quarterly correction parameters for the outer-loop adjustment mechanism of the fusion network.

[0024] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. In this invention, a dual-loop fusion mechanism significantly improves the fusion accuracy of multimodal data. The inner-loop attention mechanism can dynamically adjust weights according to the real-time features of different modalities, ensuring that key information is captured preferentially when physiological signals fluctuate abnormally or text sentiment changes abruptly. The environmental context variables introduced by the outer loop further enhance the adaptability of the fusion strategy, enabling the system to maintain stable feature extraction capabilities under different seasonal lighting conditions and other external conditions. The resulting comprehensive feature vector is closer to the actual mental health state, fundamentally improving the input quality of the early warning model.

[0025] 2. In this invention, the real-time monitoring and dynamic update mechanism effectively enhances the timeliness and personalized adaptability of the early warning system. The deployment of edge computing devices keeps model inference latency at the millisecond level. Combined with incremental learning every 24 hours, the system can quickly respond to changes in user behavior patterns. The weight adjustment strategy in the dual-loop mechanism can be automatically optimized according to individual data distribution. For example, the weight of text modality is increased for emotionally sensitive users, and the proportion of physiological signals is increased for users with large fluctuations in physiological indicators. This personalized adjustment significantly reduces the false alarm rate, making the early warning results more consistent with the actual situation of users.

[0026] 3. In this invention, the closed-loop optimization system significantly improves the stability of long-term early warning effects and the effectiveness of intervention measures. User feedback is directly converted into weight adjustment signals for the fusion network, enabling the system to continuously learn users' subjective evaluations of early warnings. Cross-period assessment incorporates long-term impacts such as seasonal changes in environmental factors into model optimization, ensuring high recognition rates across different usage stages by adjusting sensor configurations and fusion strategies. This end-to-end optimization from data collection to effect evaluation forms a virtuous cycle of adaptive evolution, significantly enhancing the initiative and accuracy of mental health interventions. Attached Figure Description

[0027] Figure 1 This is a schematic diagram illustrating the process principle of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0029] Reference Figure 1 , A real-time early warning method for mental health status based on multimodal data fusion includes the following steps: S1: Collect physiological signals such as heart rate and skin conductance through wearable devices, and simultaneously acquire behavioral data such as user social media activity, voice recordings, and mobile terminal usage time. Establish a real-time transmission channel for multi-source data to ensure that various types of data are accurately aligned on the timeline, laying the foundation for subsequent fusion analysis.

[0030] S2: The collected raw data is cleaned and denoised. An adaptive thresholding method is used to remove motion artifacts in physiological signals. Natural language processing technology is used to segment and annotate the text data. The standardized dataset output in this step will be directly used in the feature extraction stage.

[0031] S3: Construct a multi-dimensional feature space based on the preprocessed data, extract time-domain features such as heart rate variability from physiological signals, mine semantic features such as emotional polarity from text data, and optimize the feature extraction model using transfer learning to adapt to the data distribution of different users. The feature extraction results will be used as the input layer of the fusion network.

[0032] S4: Design a dual-loop fusion mechanism. The inner loop dynamically allocates the feature weights of each modality through an attention mechanism, while the outer loop introduces environmental context variables such as season and illumination to adjust the fusion strategy. The multi-dimensional features obtained in the feature extraction stage are input into the network to generate a comprehensive feature vector with context adaptability.

[0033] S5: A two-layer model consisting of an anomaly detection layer and a risk grading layer is constructed using a semi-supervised learning method. The model is trained using the comprehensive features output by the fusion network. The anomaly detection layer is responsible for identifying data patterns that deviate from the baseline, and the risk grading layer outputs warning levels based on the historical case library. The model will continuously receive feedback data from the performance evaluation stage during the training process.

[0034] S6: Deploy the trained early warning model to the edge computing device, perform real-time inference on the fusion features of new inputs, and start the incremental learning module every 24 hours to update the model parameters using the new data from the previous day, ensuring that the detection accuracy dynamically improves with the duration of use.

[0035] S7: When the model detects that the risk level exceeds the preset threshold, a multi-dimensional early warning mechanism is activated to push personalized psychological adjustment suggestions to the user. At the same time, the early warning information is encrypted and transmitted to the designated guardian's terminal. The early warning rules will be dynamically adjusted according to the intervention effect at the user feedback stage.

[0036] S8: Establish an interactive feedback interface to collect user evaluations of early warning results and the implementation status of intervention measures. Use reinforcement learning methods to transform user feedback into model optimization signals, which directly affect the weight allocation strategy of the fusion network, forming a closed-loop regulation of "early warning-intervention-feedback".

[0037] S9: Cross-period effect evaluation. A longitudinal evaluation will be conducted quarterly to compare changes in mental health indicators before and after the intervention, analyze the early warning contribution of different modalities, and feed the evaluation results back to the data collection phase to optimize sensor configuration schemes. Simultaneously, it will provide a basis for improving the situational regulation mechanism of the fusion network.

[0038] In step S1, precise timeline alignment is achieved through a three-level synchronization mechanism during multimodal data collaborative acquisition. First, at the hardware level, a combination of a GPS timing module and local clock calibration is used to configure BeiDou satellite timestamps for wearable devices, ensuring that the physiological signal sampling interval error is controlled within ±5ms. Social media data is automatically appended with UTC time generated by the server when acquired via API, and mobile terminal usage time is recorded in system-level logs with screen on / off times accurate to the second. During data transmission, an edge node caching strategy is employed. When network latency exceeds 200ms, local storage is triggered, and transmission resumes after network recovery using a timestamp comparison tool. All data must pass a timeline verification algorithm before entering the preprocessing stage. If the timestamp deviation of a certain modality exceeds a threshold, a re-sampling mechanism is initiated to ensure that physiological, textual, and behavioral data form complete samples within the same time window.

[0039] In step S2, motion artifact removal employs a combined adaptive thresholding method and wavelet transform. First, the original physiological signal undergoes a 5-level wavelet decomposition to extract high-frequency noise components. Then, the signal energy spectral density is calculated, defining frequency bands with an energy percentage below 0.01 as artifact intervals. A sliding window is then constructed, with its size dynamically adjusted based on the signal sampling rate. The standard deviation and kurtosis value are calculated for the data within the window; when the kurtosis value exceeds 3.5, the threshold sensitivity is automatically increased. During processing, the time-series characteristics of the original signal are preserved, and missing values ​​after artifact removal are filled using interpolation to ensure the preprocessed data length matches the original signal. The standardized dataset is encapsulated in JSON-LD format, with each data sample containing a modality identifier, timestamp, and feature value triplet, directly mapped to the input tensor dimension of the feature extraction model.

[0040] In step S3, the transfer learning optimization of the feature extraction model is achieved through a two-stage training process. In the first stage, a base model is pre-trained on a public dataset. Physiological signal feature extraction uses a CNN-LSTM hybrid architecture, text feature extraction uses the penultimate layer output of the BERT-base model, and behavioral features are captured using a graph neural network to capture the user interaction topology. In the second stage, personalized calibration is introduced. Data from the first two weeks of stable periods are collected to construct a private sample set. Knowledge distillation is used to transfer the feature extraction capabilities of the base model to a lightweight model, while freezing the bottom layer weights and only fine-tuning the top fully connected layer. During model optimization, the JS divergence of the feature space distribution is monitored. Transfer is stopped when the divergence value falls below 0.15, ensuring that the model can adapt to the new user data distribution while retaining its general feature extraction capabilities. The extracted multi-dimensional features are mapped to the [-1,1] interval through a normalization layer and directly used as tensor data for the input layer of the fusion network.

[0041] In step S4, the coordinated work of inner-loop attention weight allocation and outer-loop environmental regulation first involves processing the physiological signal feature set X={x1,x2,...,x...} output from the feature extraction stage. m (e.g., heart rate variability, skin conductance), text sentiment feature set Y={y1,y2,...,y n (e.g., emotional polarity, semantic intensity) and behavioral feature set Z={z1,z2,...,z...} k Inputs include (e.g., frequency of social media interaction, screen time).

[0042] The inner loop calculates the initial attention weights for each modality using a multilayer perceptron (MLP). The importance score Sx for the physiological signal modality is determined by its deviation from the historical health baseline; the text modality score Sy is based on the product of sentiment word frequency and the intensity of negative word modification; and the behavioral modality score Sz combines duration aberration and social isolation index. These scores are then input into a softmax function to obtain the inner loop's basic weights α = [α...]. x ,α y ,α z The weight values ​​are dynamically updated based on the real-time data distribution. For example, when strong negative sentiment is detected in the text data, α... y Automatically boosts by 20% to 30%. The outer loop introduces an environment context variable C={c1,c2,...,c...}. t (e.g., seasonal coefficient c) season Light intensity c light Time period c time The inner-loop weights are corrected by designing an environmental adjustment factor β. Specifically, an environmental impact matrix M ∈ Rt×3 is first constructed, where the elements m i,j This represents the influence coefficient of the i-th environmental variable on the j-th mode (e.g., winter c). seasonWhen the coefficient of influence of the physiological signal modality is 0.8, the influence coefficient m1,1 = 1.2, thus increasing its weight ratio. Then, environmental factors are transformed into a weight adjustment vector using β = sigmoid(M·α). Finally, the fusion weight w is calculated from the Hadamard product of the inner loop base weight and the outer loop adjustment vector, i.e., w = α⊙β. Based on this weight, multimodal features are linearly fused to generate a comprehensive feature vector V = w. x X+w y Y+w z Z, where w x ,w y ,w z These are the corrected weights for physiological, textual, and behavioral modalities, respectively. The formula for correcting the environmental perception attention weights is: ; w j Let be the final fusion weight for the j-th modality; S j This represents the initial importance score for the j-th modality; m i,j This represents the influence coefficient of the i-th environmental variable on the j-th mode in the environmental impact matrix. c i This is the normalized value of the i-th environmental context variable (e.g., light intensity converted to a value in the 0-1 range). t represents the total number of environment variables; k is the total number of modalities (here k=3, corresponding to physiological, text, and behavioral modalities). In step S5, the two-layer model constructed using semi-supervised learning employs a collaborative training framework. The anomaly detection layer uses the Isolation Forest algorithm, mapping the fused feature vector to a hypersphere in a high-dimensional space. The degree of deviation is determined by calculating the distance from a sample point to the center of the sphere, with the distance threshold dynamically set using the 3σ rule combined with historical user data. The risk grading layer is based on an improved K-nearest neighbor algorithm, performing cosine similarity matching between the comprehensive feature vector and risk samples in the historical case library, and selecting a weighted average of the risk levels of the top-5 similar cases as the output. During model training, 5-fold cross-validation is used. After each training round, the model parameters with the highest AUC value on the validation set are retained. Simultaneously, the intervention effectiveness rate data from the effect evaluation phase is received. When the warning accuracy for a certain type of risk falls below 85%, a retraining process for that type of sample is automatically triggered.

[0043] In step S6, the incremental learning module updates parameters using a sliding time window mechanism. The model update process starts daily at 3 AM, extracting new data from the previous 24 hours from the local database on the edge device and dividing it into incremental training and validation sets in an 8:2 ratio. Elastic Weight Consolidation (EWC) is used to protect key parameters, setting different regularization coefficients for the attention layer and environment adjustment layer of the fusion network, with the regularization strength of the attention weights being 1.5 times that of the environment adjustment factor. During the update process, the F1-score change on the validation set is monitored. If the F1-score drops by more than 5% after three consecutive updates, the system rolls back to the historical optimal parameters and adjusts the learning rate. Model deployment is accelerated using TensorRT quantization, keeping inference latency below 200ms to meet the response requirements of real-time monitoring.

[0044] In step S7, personalized psychological adjustment suggestions are generated using a hybrid strategy based on a rule engine and case-based reasoning. The rule engine has a built-in three-tiered intervention plan: for mild risk, breathing training audio is pushed (duration set to 5 or 10 minutes based on the user's historical preferences); for moderate risk, short cognitive behavioral therapy articles are added (vocabulary difficulty controlled at CEFR B1 level); and for severe risk, real-time voice guidance is triggered. The time parameters in the suggestion content are synchronized with the user's circadian rhythm, for example, pushing suggestions 30 minutes in advance based on historical data of emotional low points. The warning information is transmitted in encrypted form using the national cryptographic SM4 algorithm, and the guardian's terminal must verify it again through a dynamic password to ensure data privacy. The adjustment cycle of the warning rules is synchronized with the user feedback stage; the threshold coefficient for each risk level is recalculated every 20 valid feedback responses collected.

[0045] In step S8, the process of transforming user feedback into model optimization signals through reinforcement learning consists of four steps. First, the feedback evaluation is quantified into reward values: +1 point for "accurate warning and effective intervention," -0.5 points for "false alarm," and -0.3 points for "intervention measures not implemented." Then, a state space is constructed, containing the current fusion weight vector, environmental variable values, and the user's recent emotional trend. The value function of each action is calculated using the Q-learning algorithm, and the action space is defined as the weight adjustment amplitude (a continuous value between -0.2 and +0.2). A policy update is performed every 5 feedbacks received, directly applying the weight adjustment corresponding to the action with the largest Q value to the inner loop attention layer of the fusion network. Simultaneously, the impact of this adjustment on the subsequent warning accuracy is recorded, forming a decay coefficient matrix of the feedback signal.

[0046] In step S9, the contribution analysis of different modalities to early warning uses a combination of SHAP values ​​and confusion matrices. First, the average absolute SHAP value of each modality feature in the early warning model is calculated, with the contribution weights for physiological signals, text, and behavioral modalities denoted as Wp, Wt, and Wb, respectively. Then, a multimodal ablation experiment is constructed, and the decrease in model early warning accuracy Δp, Δt, and Δb is calculated after removing a certain modality in turn. The final contribution score is obtained by weighted summation: Score = 0.6 × SHAP value + 0.4 × accuracy decrease. In the evaluation results, if the Score of a certain modality is below 0.3 for two consecutive quarters, feedback is given to the data acquisition stage to adjust the sampling frequency of the corresponding sensor, such as reducing the acquisition interval of behavioral data or increasing the number of sensors for physiological signals. Simultaneously, the influence coefficient of environmental variables is included in the contribution analysis, providing quarterly correction parameters for the outer-loop adjustment mechanism of the fusion network.

[0047] In this invention, a dual-loop fusion mechanism significantly improves the fusion accuracy of multimodal data. The inner-loop attention mechanism dynamically adjusts weights based on the real-time features of different modalities, ensuring that key information is prioritized when physiological signals fluctuate abnormally or text sentiment changes abruptly. The environmental context variables introduced by the outer loop further enhance the adaptability of the fusion strategy, enabling the system to maintain stable feature extraction capabilities under different seasonal lighting conditions and other external factors. The resulting comprehensive feature vector more closely reflects the true mental health state, fundamentally improving the input quality of the early warning model.

[0048] In this invention, the real-time monitoring and dynamic update mechanism effectively enhances the timeliness and personalized adaptability of the early warning system. The deployment of edge computing devices keeps model inference latency at the millisecond level, and with incremental learning every 24 hours, the system can quickly respond to changes in user behavior patterns. The weight adjustment strategy in the dual-loop mechanism can be automatically optimized according to individual data distribution. For example, it can increase the weight of text modality for emotionally sensitive users and increase the proportion of physiological signals for users with large fluctuations in physiological indicators. This personalized adjustment significantly reduces the false alarm rate and makes the early warning results more consistent with the actual situation of users.

[0049] In this invention, the closed-loop optimization system significantly improves the stability of long-term early warning effects and the effectiveness of intervention measures. User feedback is directly converted into weight adjustment signals for the fusion network, enabling the system to continuously learn users' subjective evaluations of the early warnings. Cross-period assessment incorporates long-term impacts such as seasonal changes in environmental factors into model optimization, ensuring high recognition rates across different usage stages by adjusting sensor configurations and fusion strategies. This end-to-end optimization, from data collection to effect evaluation, forms a virtuous cycle of adaptive evolution, significantly enhancing the initiative and accuracy of mental health interventions.

[0050] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" or any other variations thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0051] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time early warning method for mental health status based on multimodal data fusion, characterized in that: The method includes the following steps: S1: Collect physiological signals such as heart rate and skin conductance through wearable devices, and simultaneously acquire user social media activity, voice recordings and mobile terminal usage time data to establish a real-time transmission channel for multi-source data, ensuring that various types of data are accurately aligned on the timeline, laying the foundation for subsequent fusion analysis; S2: The collected raw data is cleaned and denoised. An adaptive threshold method is used to remove motion artifacts in physiological signals. Natural language processing technology is used to segment and annotate the text data. The standardized dataset output in this step will be directly used in the feature extraction stage. S3: Construct a multi-dimensional feature space based on the preprocessed data, extract the temporal features of heart rate variability from physiological signals, mine the semantic features of emotional polarity from text data, and use transfer learning to optimize the feature extraction model so that it can adapt to the data distribution of different users. The feature extraction results will be used as the input layer of the fusion network. S4: Design a dual-loop fusion mechanism. The inner loop dynamically allocates the feature weights of each modality through an attention mechanism, while the outer loop introduces environmental context variables such as season and light intensity adjustment fusion strategies. The multi-dimensional features obtained in the feature extraction stage are input into the network to generate a comprehensive feature vector with context adaptability. S5: A two-layer model containing an anomaly detection layer and a risk classification layer is constructed using a semi-supervised learning method. The model is trained using the comprehensive features output by the fusion network. The anomaly detection layer is responsible for identifying data patterns that deviate from the baseline, and the risk classification layer outputs warning levels in combination with the historical case library. The model will continuously receive feedback data from the effect evaluation stage during the training process. S6: Deploy the trained early warning model to the edge computing device, perform real-time inference on the fusion features of new inputs, and start the incremental learning module every 24 hours to update the model parameters using the new data from the previous day, ensuring that the detection accuracy dynamically improves with the duration of use; S7: When the model detects that the risk level exceeds the preset threshold, a multi-dimensional early warning mechanism is activated to push personalized psychological adjustment suggestions to the user. At the same time, the early warning information is encrypted and transmitted to the designated guardian's terminal. The early warning rules will be dynamically adjusted according to the intervention effect of the user feedback stage. S8: Establish an interactive feedback interface to collect user evaluations of early warning results and the implementation status of intervention measures. Use reinforcement learning methods to transform user feedback into model optimization signals, which directly affect the weight allocation strategy of the fusion network and form a closed-loop adjustment. S9: Cross-cycle effect evaluation; conduct longitudinal evaluations every quarter to compare changes in mental health indicators before and after intervention, analyze the early warning contribution of different modalities, and feed the evaluation results back to the data collection stage to optimize sensor configuration schemes, while providing a basis for improving the situational regulation mechanism of the fusion network.

2. The real-time early warning method for mental health status based on multimodal data fusion as described in claim 1, characterized in that: In step S1, precise timeline alignment is achieved through a three-level synchronization mechanism during multimodal data collaborative acquisition. First, at the hardware level, a combination of GPS timing module and local clock calibration is used to configure BeiDou satellite timestamps for wearable devices, ensuring that the sampling interval error of physiological signals is controlled within ±5ms. When social media data is acquired through the API interface, the server-generated UTC time is automatically appended, and the usage time of mobile terminals is recorded in the system-level logs with screen on and off times accurate to the second. During the data transmission stage, an edge node caching strategy is adopted. When the network latency exceeds 200ms, local storage is triggered, and the data is resumed from the breakpoint after the network recovers using a timestamp comparison tool. Before all data enters the preprocessing stage, it must pass through a timeline verification algorithm. If the timestamp deviation of a certain modality exceeds the threshold, a re-sampling mechanism is initiated to ensure that physiological, text, and behavioral data form complete samples within the same time window.

3. The real-time early warning method for mental health status based on multimodal data fusion as described in claim 1, characterized in that: In step S2, motion artifact removal adopts a processing flow combining adaptive thresholding and wavelet transform. First, the original physiological signal is decomposed into 5 layers of wavelet, and after extracting the high-frequency noise component, the signal energy spectral density is calculated. The frequency band with an energy ratio of less than 0.01 is defined as the artifact interval. A sliding window is then constructed, with the window size dynamically adjusted according to the signal sampling rate. The standard deviation and kurtosis value are calculated for the data within the window, and the threshold sensitivity is automatically increased when the kurtosis value exceeds 3.

5.

4. The real-time early warning method for mental health status based on multimodal data fusion as described in claim 1, characterized in that: In step S3, the transfer learning optimization of the feature extraction model is achieved through two-stage training. In the first stage, the base model is pre-trained on a public dataset. Physiological signal feature extraction adopts a CNN-LSTM hybrid architecture, text feature extraction uses the penultimate layer output of the BERT-base model, and behavioral features are captured by a graph neural network to capture the user interaction topology. In the second stage, user personalized calibration is introduced. Data from the first two weeks of the stable period are collected to construct a private sample set. Knowledge distillation technology is used to transfer the feature extraction capability of the base model to the lightweight model, while freezing the bottom weights and only fine-tuning the top fully connected layer.

5. The real-time early warning method for mental health status based on multimodal data fusion as described in claim 1, characterized in that: In step S4, the coordinated work of inner-loop attention weight allocation and outer-loop environmental regulation first involves processing the physiological signal feature set X={x1,x2,...,x...} output from the feature extraction stage. m The text sentiment feature set Y = {y1, y2, ..., y} n } and the behavioral feature set Z={z1,z2,...,z k } as input; The inner loop calculates the initial attention weights for each modality using a multilayer perceptron. The importance score Sx for the physiological signal modality is determined by its deviation from the historical health baseline; the text modality score Sy is based on the product of emotional word frequency and the intensity of negative word modification; and the behavioral modality score Sz combines duration aberration and social isolation index. These scores are then input into a softmax function to obtain the inner loop's basic weights α = [α...]. x ,α y ,α z The weight values ​​are dynamically updated based on the real-time data distribution. For example, when strong negative sentiment is detected in the text data, α... y Automatically boosts by 20%–30%; the outer loop introduces environment context variables C={c1,c2,...,c t (e.g., seasonal coefficient c) season Light intensity c light Time period c time The inner loop weights are corrected by designing an environmental adjustment factor β; specifically, an environmental impact matrix M ∈ Rt×3 is first constructed, where the elements m i,j This represents the influence coefficient of the i-th environmental variable on the j-th mode (e.g., winter c). season When the coefficient of influence of the physiological signal modality is 0.8, the coefficient of influence m1,1 is 1.2, which enhances its weight ratio. Then, the environmental factors are transformed into a weight adjustment vector through β=sigmoid(M·α). Finally, the fusion weight w is calculated by the Hadamard product of the inner loop basic weight and the outer loop adjustment vector, i.e., w=α⊙β. Based on this weight, the multimodal features are linearly fused to generate a comprehensive feature vector V=w. x X+w y Y+w z Z, where w x ,w y ,w z These are the corrected physiological, textual, and behavioral modal weights, respectively. The formula for correcting the environmental perception attention weights is: ； w j Let be the final fusion weight for the j-th modality; S j This represents the initial importance score for the j-th modality; m i,j This represents the influence coefficient of the i-th environmental variable on the j-th mode in the environmental impact matrix. c i This is the normalized value of the i-th environment context variable; t represents the total number of environment variables; k represents the total number of modes.

6. The real-time early warning method for mental health status based on multimodal data fusion as described in claim 1, characterized in that: In step S5, the two-layer model constructed by semi-supervised learning adopts a collaborative training framework; the anomaly detection layer uses the isolated forest algorithm to map the fused feature vector to a hypersphere in a high-dimensional space, and judges the degree of deviation by calculating the distance from the sample point to the center of the sphere. The distance threshold is dynamically set by combining the 3σ rule with the user's historical data.

7. The real-time early warning method for mental health status based on multimodal data fusion as described in claim 1, characterized in that: In step S6, the incremental learning module updates parameters through a sliding time window mechanism; the model update process starts at 3:00 AM every day, extracts the new data from the previous 24 hours from the local database of the edge device, and divides it into an incremental training set and a validation set in an 8:2 ratio.

8. The real-time early warning method for mental health status based on multimodal data fusion as described in claim 1, characterized in that: In step S7, personalized psychological adjustment suggestions are generated based on a hybrid strategy of rule engine and case reasoning; the rule engine has a built-in three-level intervention plan, pushing breathing training audio for mild risk, adding cognitive behavioral therapy articles for moderate risk, and triggering real-time voice guidance for severe risk.

9. The real-time early warning method for mental health status based on multimodal data fusion as described in claim 1, characterized in that: In step S8, the process of transforming user feedback into model optimization signals through reinforcement learning is divided into four steps: First, the feedback evaluation is quantified into reward values, with "accurate warning and effective intervention" scoring +1 point, "false alarm" scoring -0.5 points, and "intervention measures not implemented" scoring -0.3 points; then, a state space is constructed, which includes the current fusion weight vector, environmental variable values, and the user's recent emotional trend. In step S9, the contribution analysis of different modal data to early warning uses a combination of SHAP value and confusion matrix. First, the average absolute value of SHAP for each modality feature in the early warning model is calculated, and the contribution weights of physiological signals, text, and behavioral modalities are denoted as Wp, Wt, and Wb, respectively. Then, a multimodal ablation experiment is constructed, and the decrease in model early warning accuracy Δp, Δt, and Δb is calculated after removing a certain type of modality in turn. The final contribution score is obtained by weighted summation: Score = 0.6 × SHAP value + 0.4 × decrease in accuracy. In the evaluation results, if the Score of a certain modality is lower than 0.3 for two consecutive quarters, the sampling frequency of the corresponding sensor is adjusted in the data acquisition stage, such as reducing the acquisition interval of behavioral data or increasing the number of sensors for physiological signals. At the same time, the influence coefficient of environmental variables is included in the contribution analysis to provide quarterly correction parameters for the outer-loop adjustment mechanism of the fusion network.

10. A real-time early warning device for mental health status based on multimodal data fusion, characterized in that: The device is internally equipped with a program that runs the real-time early warning method for mental health status based on multimodal data fusion as described in any one of claims 1 to 9.

Images