An intelligent early warning system for mental health risks based on multi-modal data
A multimodal mental health monitoring system that utilizes localized processing and federated learning addresses data privacy and model generalization issues, enables personalized and interpretable risk warnings, dynamically adjusts resource usage, and supports closed-loop monitoring with both automated and manual interventions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Filing Date
- 2026-03-09
- Publication Date
- 2026-07-14
Smart Images
Figure CN122392910A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer data processing and artificial intelligence technology, specifically to an intelligent early warning system for mental health risks based on multimodal data. Background Technology
[0002] With the widespread adoption of wearable devices and smart terminals, monitoring users' mental health using multimodal data collected by these devices (such as heart rate variability, sleep patterns, and frequency of social interactions) has become an important research direction in the field of public health. Through long-term analysis of passively perceived data, early signs of mental health problems such as depression and anxiety can be identified, enabling early intervention.
[0003] However, existing mental health monitoring technologies still face numerous challenges in practical applications. Most current mainstream solutions employ a centralized cloud computing architecture, requiring users to upload raw physiological signals and behavioral data collected from their devices to a cloud server for unified processing and model inference. Because mental health data involves extremely high levels of personal privacy, this method of uploading and centrally storing raw data poses a serious risk of data leakage, leading to decreased user trust in the system and even refusal to authorize data collection. While some technologies attempt to introduce localized processing, the generalization ability of local models is often difficult to guarantee without large-scale cloud-based data collaboration.
[0004] At the model training level, existing deep learning algorithms typically rely on large amounts of high-quality labeled data (i.e., supervised learning). In the field of mental health, obtaining accurate labels often requires professional clinical diagnosis or frequent subjective user questionnaires. The former is costly and difficult to sustain, while the latter can easily disrupt users' normal lives, leading to user fatigue and decreased compliance. Simply relying on fixed expert rules for judgment is also difficult to adapt to the physiological and behavioral differences among individual users, resulting in a high false positive rate.
[0005] Furthermore, existing risk prediction models are mostly black-box structures, only outputting a risk probability value without providing an explanation of the specific causes of the risk. For psychiatrists or users, predictions lacking interpretability are difficult to use as effective evidence for clinical diagnosis or self-regulation. Meanwhile, mobile devices have limited computing resources and battery life, and existing monitoring systems often use fixed sampling frequencies. While high-frequency sampling can capture subtle changes, it leads to excessively rapid power consumption, while low-frequency sampling, although energy-efficient, is prone to missing sudden psychological crisis events, making it difficult to strike a balance between monitoring accuracy and system power consumption. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention provides an intelligent early warning system for mental health risks based on multimodal data, which solves the problems of discontinuous control trajectory, static and unadjustable emotion mapping, lack of user feedback loop, weak adaptive ability, and insufficient fusion of multidimensional control information in existing technologies.
[0007] To achieve the above objectives, the present invention provides the following technical solution: This invention provides an intelligent early warning system for mental health risks based on multimodal data. The system mainly consists of client terminals and a central server connected via a communication network. At the system architecture level, this invention decentralizes the collection, processing, model training, and inference of sensitive data to the user-side client terminals, while the central server is only responsible for aggregating and updating global model parameters, thereby achieving source protection of data privacy.
[0008] In the specific operational logic of the client terminal, the system first collects users' physiological signal data (such as photoplethysmography pulse wave and skin conductance response), behavioral trajectory data (such as GPS location and acceleration), and social interaction data through a multimodal perception module. The data processing module then performs feature extraction on this heterogeneous data. Specifically, for physiological signal data, it uses Fast Fourier Transform to extract the frequency domain features of heart rate variability; for behavioral trajectory data, it calculates the trajectory entropy of the location coordinate sequence and the average daily sleep duration; and for social interaction data, it statistically analyzes application interaction frequency and voice features. Subsequently, the system employs a combination of high-frequency signal mean pooling downsampling and sparse data linear interpolation upsampling to address the issue of inconsistent sampling frequencies in multimodal data. This aligns various features to a unified time granularity and performs normalization processing, generating a multimodal feature fusion vector.
[0009] To address the issue of traditional deep learning relying on large amounts of manually labeled data, this invention designs a dynamic label construction module. This module employs a two-layer label generation mechanism combining heuristic rules and active learning. On one hand, based on a preset expert rule engine (including discriminant functions for abnormal physiological stress, behavioral withdrawal, and social isolation), when heart rate variability, sleep duration, or interaction frequency meets preset threshold conditions, weakly supervised pseudo-labels are automatically generated to solve the cold start problem. On the other hand, the system calculates the information entropy of the model's predicted probability distribution. When the information entropy exceeds a query threshold, it indicates that the model has uncertainty about the current state, triggering an active query mechanism to obtain the user's real feedback labels. Finally, based on whether real feedback has been obtained, the system dynamically fuses real labels and pseudo-labels to generate hybrid training labels for continuous model iteration.
[0010] Regarding model training and privacy protection, the local training module constructs a Long Short-Term Memory (LSTM) network model and performs local training using multimodal feature fusion vectors and mixed training labels. To prevent the model from remembering private details in the training data, the system introduces a differential privacy protection mechanism. Before uploading gradients, the system first calculates the L2 norm of the original gradients and performs gradient clipping. Then, based on the privacy budget parameters, it injects a noise vector following a multidimensional Gaussian distribution to generate encrypted noise gradients. The global aggregation module on the central server receives the encrypted noise gradients from each client, executes a federated average aggregation algorithm to update the global model parameters, and distributes the updated parameters to overwrite the local models, achieving model optimization through multi-party collaboration.
[0011] To enhance the transparency and credibility of the system, this invention includes an interpretable evaluation module. This module utilizes the integral gradient algorithm to calculate the cumulative effect of the model output gradient relative to changes in input features as the input changes from the baseline input to the current input. This quantifies the contribution of each feature dimension and generates explanations for the contributions of key features, thus achieving a shift from black-box prediction to interpretable attribution.
[0012] Furthermore, this invention constructs an adaptive feedback control closed loop. The feedback control module dynamically adjusts the system's operating parameters based on the risk prediction probability output by the model. When the risk is low, the system maintains a baseline sampling frequency to reduce energy consumption; when the risk enters a high-risk range, the system automatically increases the sampling frequency of the multimodal perception module to capture subtle features, while simultaneously lowering the query threshold of the dynamic tag construction module to increase the frequency of proactive inquiries. If the high-risk state persists, the system packages an explanatory report and trend chart, encrypts it, and pushes it to the monitoring terminal, completing the closed loop from automatic monitoring to manual intervention.
[0013] This invention provides an intelligent early warning system for mental health risks based on multimodal data. It has the following beneficial effects: 1. This invention achieves strict data privacy protection through the collaboration of a local training module and a global aggregation module. The system utilizes a differential privacy mechanism, uploading only encrypted noise gradients generated through gradient clipping and Gaussian noise injection to the central server. Raw multimodal data involving user privacy, such as physiological signals, behavioral trajectories, and social interactions, are always processed locally on the client terminal. This effectively blocks the leakage paths of sensitive data during network transmission and cloud storage, resolving the conflict between data utilization and privacy security in mental health monitoring.
[0014] 2. This invention addresses the challenge of scarce high-quality labeled samples in mental health model training by utilizing a dynamic labeling module. By combining weakly supervised pseudo-labels generated by expert rules with an active query mechanism triggered by information entropy, the system can achieve cold start of the model without human intervention. It also selectively acquires high-value, genuine user feedback labels only when the model's predictions are uncertain, reducing reliance on manual labeling while ensuring continuous model optimization as user data accumulates, thus improving adaptability to individual differences.
[0015] 3. This invention constructs an adaptive monitoring closed loop that balances resources and accuracy through a feedback control module. Based on real-time calculated risk prediction probabilities, the system dynamically adjusts the sampling frequency of the multimodal sensing module and the trigger threshold for active queries. During low-risk periods, the sampling frequency is reduced to conserve terminal computing resources and power, while during high-risk periods, the data acquisition density is automatically increased to capture subtle pathological features. Combined with the key feature contribution interpretation output by the interpretive evaluation module, the system not only achieves accurate early warning but also provides intuitive attribution evidence for subsequent manual intervention. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the overall system architecture and module composition of the present invention; Figure 2 This is a flowchart of the multimodal data hierarchical sensing and spatiotemporal alignment process of the present invention; Figure 3 This is a flowchart of the training process for the federated LSTM model of the present invention; Figure 4 This is a schematic diagram of the adaptive monitoring strategy and dynamic feedback closed-loop principle of the present invention; Figure 5 Simulation of multimodal data temporal variation and risk probability evolution for this invention Figure 1 ; Figure 6 Simulation of multimodal data temporal variation and risk probability evolution for this invention Figure 2 ; Figure 7 Simulation of multimodal data temporal variation and risk probability evolution for this invention Figure 3 .
[0017] Among them, 100 is the client terminal; 101 is the multimodal perception module; 102 is the data processing module; 103 is the dynamic label construction module; 104 is the local training module; 105 is the interpretive evaluation module; 106 is the feedback control module; 200 is the central server; 201 is the global aggregation module; and 300 is the communication network. Detailed Implementation
[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0019] See attached document Figure 1 This invention provides an intelligent early warning system for mental health risks based on multimodal data. The system includes multiple client terminals 100, a central server 200, and a communication network 300. The multiple client terminals 100 establish communication connections with the central server 200 through the communication network 300. The system is configured to train and infer a distributed mental health risk assessment model using multi-source heterogeneous data, while protecting the privacy of users' original data.
[0020] The client terminal 100 is a computing device deployed on the user side, including but not limited to smartphones, wearable devices, or personal computers. The client terminal 100 internally includes: a multimodal perception module 101, a data processing module 102, a dynamic label construction module 103, a local training module 104, an interpretability evaluation module 105, and a feedback control module 106. These modules transmit signals and exchange data via an internal bus or data interface.
[0021] The multimodal sensing module 101 is configured to connect to and control a physical sensor array and an application programming interface (API) for collecting multi-dimensional state data of the user. The data collected by the multimodal sensing module 101 includes physiological surrogate index data. behavioral trajectory feature data And social and emotional characteristics data Among them, the physiological substitute index data The data originates from photoplethysmography (PPG) sensors and skin conductance sensors, including heart rate variability and skin conductance values; the behavioral trajectory feature data... The data originates from a GPS receiver and an accelerometer, including location movement data and motion intensity data; the social and emotional feature data... Applications originating from the terminal use logs and audio input interfaces, including interaction frequencies and speech acoustic characteristics.
[0022] The data processing module 102 is connected to the multimodal sensing module 101 and is configured to receive the physiological surrogate index data. The behavioral trajectory feature data and the aforementioned social and emotional feature data The data processing module 102 performs a spatiotemporal alignment operation on the heterogeneous data with different sampling frequencies. The spatiotemporal alignment operation includes mean pooling of the high-frequency continuous signal and interpolation of the low-frequency discrete signal. The data processing module 102 further standardizes the aligned data to generate data corresponding to the time step. Multimodal feature fusion vector The multimodal feature fusion vector It is transmitted to the local training module 104 and the dynamic label construction module 103.
[0023] The dynamic label construction module 103 is configured to generate supervision signals for model training. The dynamic label construction module 103 has a pre-built heuristic rule base, which is based on the multimodal feature fusion vector. Calculate weakly supervised pseudo-labels The dynamic label construction module 103 is also configured to calculate the uncertainty entropy value of the current model prediction. When the uncertainty entropy value exceeds a preset threshold, an interactive interface is triggered to obtain the user's input of real feedback labels. The dynamic label construction module 103 is based on the weakly supervised pseudo-labels. and the aforementioned real feedback tags Generate mixed training labels and the mixed training labels The data is transmitted to the local training module 104.
[0024] The local training module 104 is connected to both the data processing module 102 and the dynamic label construction module 103. The local training module 104 stores a local model based on a Long Short-Term Memory (LSTM) network. The local training module 104 utilizes the multimodal feature fusion vector... As input, the hybrid training labels are used As the target value, forward propagation is performed to compute the predicted probability distribution, and backpropagation is performed to compute the gradients of the model parameters. .
[0025] The local training module 104 is further configured to perform differential privacy processing. The local training module 104 processes the gradients... Perform L2 norm clipping and Gaussian noise injection to generate encrypted noise gradients. The local training module 104 transmits the encrypted noise gradient through the communication network 300. Send to the central server 200, and receive the updated global model parameters issued by the central server 200. .
[0026] The central server 200 includes a global aggregation module 201. The global aggregation module 201 is configured to receive the encrypted noise gradient from the plurality of client terminals 100. The global aggregation module 201 uses a federated averaging algorithm to weight and aggregate the received gradients, update the global model parameters, and distribute the updated parameters back to the multiple client terminals 100 through the communication network 300 to complete one round of federated learning iteration. The interpretability evaluation module 105 is connected to the local training module 104. The interpretability evaluation module 105 is configured to calculate the multimodal feature fusion vector locally on the client terminal 100, based on the trained local model parameters, using an integral gradient algorithm. The contribution value of each dimension of features to the risk prediction results is calculated. The interpretive evaluation module 105 generates structured data containing key risk features based on the contribution values, which is used to present risk attribution on the display interface.
[0027] The feedback control module 106 is connected to both the local training module 104 and the multimodal perception module 101. The feedback control module 106 is configured to receive the risk prediction probability output by the local training module 104. The feedback control module 106 will determine the risk prediction probability. The risk level is compared with a preset risk classification threshold. Based on the comparison result, the feedback control module 106 generates a sampling control signal and sends it to the multimodal sensing module 101 to dynamically adjust the sampling frequency of the physical sensor array. When the risk prediction probability... When the risk level is in a high-risk range, the feedback control module 106 increases the sampling frequency; when the risk prediction probability... When the risk level is low, the feedback control module 106 reduces the sampling frequency.
[0028] See attached document Figure 1 and attached Figure 2 The multimodal perception module 101 is specifically configured to execute a hierarchical perception strategy, which divides the data source into three dimensions: physiological substitute indicators, behavioral trajectory features, and social and emotional features. The multimodal perception module 101 internally includes a physiological signal acquisition unit, a behavioral sensor interface unit, and an application log parsing unit.
[0029] The physiological signal acquisition unit is connected to the photoplethysmography (PPG) sensor and the skin conductance response (GSR) sensor via a physical contact interface. This is for the physiological surrogate index data. The system is configured to first acquire raw pulse wave signals using a PPG sensor at a preset sampling rate (e.g., 25Hz to 100Hz), and detect signal peaks to calculate adjacent RR intervals. The data processing module 102 performs power spectral density analysis on the RR interval sequence, converting the time-domain signal to a frequency-domain signal using a Fast Fourier Transform (FFT). Specifically, the system extracts low-frequency power... (Frequency range 0.04Hz-0.15Hz) and high-frequency power (Frequency range 0.15Hz-0.4Hz). Given the hardware limitations of directly measuring biochemical indicators such as cortisol, this invention utilizes the balance state of the autonomic nervous system as a proxy indicator of physiological stress to calculate the frequency domain characteristics of heart rate variability. As a key parameter for measuring the balance between sympathetic and parasympathetic nerve tension, its calculation formula is expressed as follows: ; in, Represents the power spectral density function. The frequency is used. Simultaneously, the physiological signal acquisition unit collects the amplitude and frequency of skin conductance level (SCL) and skin conductance response (SCR) via a GSR sensor to characterize the user's emotional arousal. The physiological surrogate index data... Ultimately, it was constructed to include normalization. A vector of average heart rate, mean SCL, and SCR frequency.
[0030] The behavior sensor interface unit communicates with the Global Positioning System (GPS) and Inertial Measurement Unit (IMU) built into the client terminal 100. This is for the behavior trajectory feature data. The system does not directly record absolute latitude and longitude coordinates to protect privacy. Instead, it clusters the user's geographic location coordinates into a finite set of rest points. The system quantifies the regularity of a user's lifestyle and the richness of their activity range by calculating the information entropy of their location trajectories. For a location sequence within a preset time window, it calculates the information entropy of each stop point. Access probability Thus, the trajectory entropy is obtained. : ; when When the numerical value decreases, it indicates a contraction in the user's activity range, serving as a behavioral characteristic input for depressive mood. Furthermore, the behavior sensor interface unit synchronously acquires triaxial acceleration data output from the IMU, extracts the user's daily step count and duration of high-intensity activity by calculating the integral of the signal vector magnitude (SVM), and incorporates this data into the behavioral trajectory feature data after Z-score normalization. .
[0031] The application log parsing unit is configured to read application usage records and audio input streams via system-level APIs to obtain the social and emotional feature data. When processing audio data, the system extracts only non-semantic acoustic features to avoid content privacy risks. Specifically, the system pre-emphasizes, frames, and windowes the speech segments captured by the microphone, extracts Mel-frequency cepstral coefficients as the basic acoustic features, and calculates fundamental frequency perturbation and amplitude perturbation indices. The fundamental frequency perturbation... It is used to reflect the periodic stability of vocal cord vibration, and its calculation is based on the period of adjacent fundamental tones. The difference: ; in The total number of extracted pitch cycles. Simultaneously, the application log analysis unit statistically analyzes the number of screen wake-ups, single session duration, and backspace key usage frequency of the instant messaging software. The backspace key usage frequency serves as a proxy indicator of cognitive slowness and hesitation. The aforementioned acoustic features and interactive behavior statistics together constitute the social and emotional feature data. , and the physiological alternative index data and behavioral trajectory feature data Keep timestamps synchronized and transmit them to subsequent modules for fusion processing.
[0032] The data processing module 102 is configured to perform data cleaning and standardization processes to address the asynchronous nature of different modalities in the time dimension and the dimensional differences in the numerical dimension. The data processing module 102 first defines a unified time step. (In this embodiment, it is preferably set to 60 minutes) as the smallest granularity for data aggregation, and a continuous time window sequence is divided on the time axis. .
[0033] Regarding the aforementioned physiological alternative indicator data For high-frequency continuous signals, the data processing module 102 employs a mean pooling strategy for downsampling. Specifically, for any time window... Internally collected Physiological characteristic sampling points The system calculates its arithmetic mean as the representative value for that time step. If a sensor detachment causes signal loss within a certain time window, the system is configured to use the value from the previous time window. The values are filled with zero-order hold padding to maintain the continuity of the timing sequence.
[0034] Regarding the behavioral trajectory feature data and the aforementioned social and emotional feature data For low-frequency or sparse discrete signals present in the data (e.g., sleep quality scores generated only once a day or social interaction events occurring intermittently), the data processing module 102 performs time-weighted resampling. For discrete event statistics data (e.g., interaction frequency), the system calculates the cumulative frequency of events within the time window. For non-periodic sparse state data, the system uses a linear interpolation algorithm to map the sparse data to a uniform time step. Assuming at time point... and The numerical values were observed respectively and For target time steps between the two, The interpolation calculation formula is as follows: ; Through the aforementioned downsampling and upsampling operations, the system aligns the originally asynchronous multimodal data into feature sequences with the same time index. After completing the spatiotemporal alignment, the data processing module 102 further performs dynamic normalization processing on the feature data based on user historical statistics. To eliminate the dimensional differences between different features (such as millisecond-level values of heart rate variability and thousand-level values of step count), while preserving the individual user's trend relative to their baseline, the system maintains a length of... A historical sliding window (e.g., data from the past 7 days). For any aligned features Using the mean within the sliding window with standard deviation Perform Z-score normalization. The transformed features... The calculation is as follows: ; in, To prevent small constants with a denominator of zero (e.g.) The normalized features of each dimension are then concatenated sequentially to construct the generation time. Comprehensive feature vector This vector It not only contains the current instantaneous state information, but also implies the degree of deviation relative to the user's individual historical benchmark, and is then transmitted to the dynamic label construction module 103 and the local training module 104 for subsequent processing.
[0035] The dynamic label construction module 103 is internally configured with an expert rule engine, which pre-stores multiple sets of digital discrimination logics based on clinical psychology diagnostic criteria (such as DSM-5). The dynamic label construction module 103 is configured to utilize the multimodal feature fusion vector without requiring manual user input. Automatically generate basic supervision signals for model training, i.e., weakly supervised pseudo-labels. .
[0036] The expert rule engine first processes the input multimodal feature fusion vector. Feature decoupling is performed to extract component values of key risk dimensions, and single-item anomaly detection based on statistical thresholds is executed. The system defines discriminant functions for three core dimensions: physiological stress anomaly discriminant function. Behavioral withdrawal discriminant function and social distancing discriminant function .
[0037] For the physiological stress dimension, the system reads the normalized heart rate variability frequency domain features. When this value is higher than the user's historical baseline (e.g., exceeding the historical mean by 2 standard deviations), it indicates that the sympathetic nervous system is overactive. The aforementioned physiological stress anomaly discrimination function... The calculation logic is as follows: ; in, This is an indicator function that outputs 1 when the condition is met, and 0 otherwise. and These are the mean and standard deviation of the user's historical data, respectively. This is a preset sensitivity coefficient (for example, a value of 1.96, corresponding to a 95% confidence interval).
[0038] For behavioral and social dimensions, the system focuses on monitoring joint anomalies in sleep patterns and social interactions. The system extracts average daily sleep duration from behavioral characteristics. With activity entropy Interaction frequency is analyzed from social features. The expert rule engine performs multi-condition joint logical operations, triggering high-risk pseudo-labels only when physiological indicators are abnormal and accompanied by behavioral changes, thus reducing the false alarm rate. The weakly supervised pseudo-labels... The generation logic is expressed as follows: ; in, This represents the logical AND operation. Represents a logical OR operation; The sleep deprivation threshold (e.g., less than 5 hours), The social isolation threshold (e.g., fewer than 3 interactions per day).
[0039] Through the above logic, the dynamic label construction module 103 transforms the continuously input unlabeled multimodal data stream into training sample pairs with binarized or probabilistic labels in real time. This mechanism ensures that even during the cold start phase, when there is a lack of professional medical diagnosis or active user feedback, the local training module 104 can still obtain continuous gradient update signals, thereby maintaining the long short-term memory network's continuous tracking and learning of the user's psychological state evolution trend. The generated weakly supervised pseudo-labels... It is then passed to subsequent processing units as the default target value until it is overridden or corrected by user feedback tags with higher confidence.
[0040] In order to calibrate the weakly supervised pseudo-labels To address existing noise and resolve the discrimination ambiguity issue in the model under critical conditions, the dynamic label construction module 103 is further configured with an active learning unit. This unit does not request user feedback on data at all time steps, but instead performs sparse sampling based on the uncertainty of the model's predictions.
[0041] The active learning unit first obtains the predicted probability distribution output by the local training module 104 at the previous time step. It assumes the model is familiar with the current input features. Belongs to the risk category ( The predicted probabilities (representing low risk and high risk respectively) are: The system calculates the Shannon entropy of this probability distribution. As a quantitative indicator for measuring the cognitive uncertainty of a model. The calculation formula is as follows: ; when The larger the value, the closer the model's judgment of the current sample is to random guessing (e.g., the probability is close to 0.5), that is, it is in the fuzzy zone of the decision boundary.
[0042] The active learning unit will calculate the entropy value. With preset query threshold Perform a comparison. Only if the conditions are met. When the time interval between the current moment and the last triggered query exceeds a set silent period (e.g., 4 hours), the system generates an active query command. This command triggers a micro-interactive window to pop up on the display interface of the client terminal 100, requesting the user to self-assess their current psychological stress level (e.g., using a visual 1-5 Likert scale or slider). The values entered by the user in the interactive interface are mapped to normalized real feedback labels. (Value range [0,1]).
[0043] After receiving user feedback, the dynamic label building module 103 executes a label fusion strategy to generate hybrid training labels for the final model backpropagation. This strategy prioritizes user feedback with absolute confidence, while reverting to weakly supervised pseudo-labels during periods lacking user feedback. The final label determination logic is defined as follows: ; in, As an indicator variable, when the system successfully obtains user feedback hour ,otherwise Through the above mechanism, the system constructs a hybrid dataset containing a large number of automatically labeled samples and a small number of high-confidence manually calibrated samples. The local training module 104 trains the dataset based on this hybrid label. The loss function is calculated to achieve personalized convergence of the model's discriminative ability while protecting privacy.
[0044] See attached document Figure 3 The local training module 104 is configured to build and run a Long Short-Term Memory (LSTM) network in the client's local environment to perform time series data modeling and parameter updates, without needing to fuse the original multimodal feature vectors throughout the process. The data is transmitted to the local device. The local training module 104 includes an input mapping layer, an LSTM recurrent layer, and a classification output layer. At time steps... The LSTM recurrent layer receives the feature input at the current time step through its internal forget gate, input gate, and output gate mechanism. Compared to the previous hidden state Dynamically update cell state This is to store long-term dependent information. Finally, the model outputs the predicted probability distribution of mental health risk at the current moment through the Softmax activation function. And combined with the hybrid training labels generated in the aforementioned steps Calculate the value of the binary cross-entropy loss function.
[0045] After completing the forward propagation and loss calculation, the local training module 104 executes the backpropagation algorithm to calculate the local model parameters. relative to the original gradient vector of the loss function To defend against model inversion attacks and meet the security requirements of differential privacy, the system performs gradient perturbation before gradient uploading. First, the local training module 104 performs gradient pruning to limit the influence boundary of a single data sample on the global model update, thereby constraining sensitivity. The system presets a pruning threshold for the gradient L2 norm. The original gradient is calculated according to the following formula. Convert to clipping gradient ; This operation ensures that the magnitude of all gradient vectors involved in training is constrained to a certain value. Within this range. Subsequently, the local training module 104 applies a Gaussian mechanism to inject noise into the pruned gradient. The system then adjusts the gradient according to the globally set privacy budget parameters. With the corresponding noise multiplier The generated data follows a mean of 0 and a variance of 0. Multidimensional Gaussian distribution noise vector The noise vector is then superimposed onto the clipping gradient to generate the final encrypted noise gradient to be uploaded. : ; Through this step, the system mathematically guarantees that the gradient of the output satisfies... Differential privacy constraints prevent attackers from inferring the specific physiological or behavioral characteristics of a predefined user from gradient values.
[0046] The global aggregation module 201 in the central server 200 receives data from [unclear] via a secure encrypted channel. Encryption noise gradient of 100 participating client terminals The global aggregation module 201 does not directly access the local datasets of each client, but instead performs weighted aggregation of the received gradient updates based on a federated averaging algorithm. Let the... The number of samples used by each client in training is The total sample size is Global model parameters The update rules are as follows: ; in The global learning rate is used. Because Gaussian noise has additive homomorphic properties, after aggregating gradients from a large number of clients, independently injected noise will statistically partially cancel each other out. This ensures individual privacy while making the aggregated global gradient direction approximate the true gradient direction, guaranteeing that the global model converges to an effective optimal solution for risk identification. Updated global parameters. It is then distributed back to each client terminal 100 to overwrite the local model parameters and start the next round of monitoring and training iterations.
[0047] See attached document Figure 4 The interpretative evaluation module 105 is configured to perform visual attribution of model decisions locally on the client terminal 100, aiming to reveal the causes leading to preset time steps. The specific characteristics that lead to an increased probability of predicting mental health risks are identified. The interpretive assessment module 105 does not rely on general global feature importance, but rather on the specific characteristics of a single user's current input instance. Calculate the feature contribution at the instance level.
[0048] The interpretive evaluation module 105 first defines a baseline input vector. This vector represents a baseline of a non-informative or neutral mental state. In this embodiment, It is set as either a vector of all zeros or the mean vector of the user's historical data. This is for calculating input features. The Middle Features in each dimension Output results of the model To assess the contribution of the baseline input, the system employs the integral gradient algorithm. This algorithm examines the contribution from the baseline input. Change along a linear path to the current input During the process, the cumulative effect of the model output gradient relative to the input changes.
[0049] In the specific calculation process, since the computer cannot handle continuous integrals, the interpretative evaluation module 105 uses the Riemann sum approximation method for discretization calculation. The system sets the number of interpolation steps. (For example ),generate The intermediate interpolation sample. For the first... step( ), calculate interpolation points When input is fed into the LSTM model in the local training module 104, the model outputs the gradient relative to the input features. Finally, the... Attribution values of dimensional features Calculated using the following formula ; in, Represents the model prediction function For input features The partial derivatives. This calculation process runs entirely on the local processor of the client terminal 100, without requiring the input data or intermediate gradients to be uploaded to the server, thus ensuring the privacy and security of the interpretability analysis process.
[0050] After obtaining the original attribution value vector along the feature dimensions, the interpretative evaluation module 105 performs... Take the absolute value and sort in descending order to select the one with the highest contribution. Several key features. Subsequently, the system calls a pre-defined semantic mapping table to convert the abstract feature dimension names into clinically understandable natural language descriptions. For example, if corresponding to... Dimensional The highest value and a positive contribution are mapped by the system to an autonomic nervous system regulatory imbalance; if the corresponding value is... (Trajectory entropy) dimension A higher value indicates a decrease in the regularity of daily life.
[0051] Finally, the interpretive assessment module 105 generates structured interpretive data containing key risk factors and their contribution weights. This data is transmitted to the user interface of the client terminal 100 and presented in the form of visual charts (such as radar charts or bar charts) combined with text summaries. When the system is in a high-risk warning state, this structured interpretive data is encrypted and packaged, and sent to the medical institution along with the warning signal, serving as an auxiliary decision-making basis for doctors to conduct clinical interventions, thus realizing the logical support from black-box prediction to white-box diagnosis.
[0052] The feedback control module 106 constructs a closed-loop control circuit connecting the model prediction results and the front-end sensing behavior. It is configured to dynamically adjust the system's operating parameters based on the real-time assessed risk level, achieving an automated balance between monitoring accuracy and terminal energy consumption. The feedback control module 106 continuously receives the latest time-step risk prediction probability value output by the local training module 104. And input it into the internal risk classification comparator.
[0053] The risk grading comparator is pre-set with a set of grading thresholds, including low-risk thresholds. With high risk threshold The system is based on The numerical values divide the current user status into three levels: a safe zone, a warning zone, and a high-risk zone. The feedback control module 106 generates corresponding hardware control commands and software interaction commands based on the judgment results. When the system determines that the area is within a safe range, the feedback control module 106 generates an energy-saving mode command; when When the system determines that the area is high-risk, the feedback control module 106 generates a deep monitoring mode command.
[0054] For hardware-level feedback adjustment, the feedback control module 106 establishes a bidirectional control link with the multimodal sensing module 101. Upon receiving a deep monitoring mode command, the multimodal sensing module 101 performs a sampling frequency doubling operation. Specifically, the system increases the activation frequency of physiological signal acquisition units (such as PPG and GSR sensors). From the reference frequency (For example, once per hour) Adjust to a high-frequency state. The adjustment of the sampling frequency follows a linear growth function or step function based on the risk probability, and its control logic is as follows: ; in, This represents the frequency gain coefficient. By increasing the sampling frequency, the system can capture more fine-grained physiological fluctuations under high-risk conditions, such as the sharp decline in heart rate variability over a short period, thus providing higher-resolution input data for model inference at the next time step. .
[0055] Regarding software-level feedback adjustment, the feedback control module 106 synchronously sends a sensitivity adjustment signal to the dynamic tag construction module 103. When in a high-risk zone, the system automatically lowers the uncertainty entropy threshold in the active query mechanism. This makes it easier for the system to trigger the micro-interaction inquiry window. This mechanism ensures that the system can promptly obtain the user's true subjective feedback at critical moments when the user's psychological state is abnormal. This real-world feedback is not only used for immediate risk assessment, but also added to the training set as a high-confidence, strong supervisory signal. It is then used to quickly calibrate the weights of the local LSTM model via backpropagation, correcting any biases in the model.
[0056] Furthermore, the feedback control module 106 is also configured to trigger a tiered intervention mechanism. When the risk remains in the high-risk range for more than a preset duration threshold (e.g., three consecutive time windows), the system generates an alarm data packet. This data packet contains the current risk probability, explanations of key pathogenic characteristics generated by the interpretive assessment module 105 (e.g., severe sleep deprivation and persistently elevated stress levels), and a trend graph for the recent period. With prior user authorization, the feedback control module 106 encrypts and pushes the alarm data packet to a designated guardian's terminal or medical institution server via the communication network 300, thereby completing a closed-loop process from automatic monitoring to manual intervention. This dynamic adjustment strategy ensures that the system remains silent and consumes little power under normal conditions, while instantly switching to a high-performance monitoring state when an anomaly occurs.
[0057] See attached document Figure 5 -Appendix Figure 7 This figure details the dynamic evolution of physiological indicators, behavioral characteristics, and risk prediction probabilities for a preset user over a 336-hour (14-day) monitoring period. The horizontal axis represents monitoring time (hours), and the vertical axis represents the normalized indicator values. Figure 5 As shown in Figure 6, during the initial phase of the monitoring period (T=0 hours to T=192 hours, i.e., day 0 to day 8), the user is in a normal daily routine. At this time, the heart rate variability frequency domain characteristic (LF / HF ratio) collected by the multimodal sensing module fluctuates slightly around the baseline value of 2.0, consistently remaining below the preset physiological warning threshold of 3.0 (e.g., ...). Figure 5 (As shown by the dashed line). Meanwhile, the user's average daily sleep duration remained around 7.5 hours, exceeding the sleep deprivation threshold of 5.0 hours (as shown by the dashed line). Figure 6 (As shown by the midpoint line). During this period, the local training module outputs the probability of mental health risk. The system remains stable within a safe range of around 0.15, maintaining a low-power silent operation mode.
[0058] The monitoring data showed a changing trend starting from T=192 hours. Figure 5 The LF / HF ratio gradually increased and exceeded the 3.0 threshold at around T=210 hours, indicating increased sympathetic tone. Figure 6 The data showed a rapid decline in average daily sleep duration, falling below the 5.0-hour threshold and reaching a low of around 4.0 hours. This combination of features triggered the weakly supervised rules of the dynamic labeling module.
[0059] Key Event Node 1: Proactive Feedback Calibration (t=216 hours) like Figure 7 As indicated by the circular marker, at T=216 hours (i.e., day 9), the model detects that the input features are in the uncertainty region of the decision boundary (at this time...). It is approximately 0.6, and the information entropy is... (At a relatively high level), the entropy weight active query mechanism is triggered. The system pops up a micro-interaction window on the client terminal, where the user provides subjective feedback under high pressure. This real label is immediately incorporated into the training, calibrating the local model and causing the subsequent risk probability curve to show a steeper upward trend, reflecting the model's increased confidence in risk confirmation.
[0060] Key Event Node Two: High-Risk Early Warning and Closed-Loop Intervention (t=240 hours) As the stress state continues, at T=240 hours (i.e., day 10), such as Figure 7 The risk probability output by the system is indicated by the triangle marking. It climbed to around 0.9, officially breaking through the preset high-risk threshold. ( Figure 7 (Middle dashed line). At this time, the feedback control module immediately performs closed-loop intervention: on the one hand, it generates an attribution explanation report containing abnormal physiological indicators (LF / HF>3.0) and severe sleep deprivation and pushes it to the monitoring terminal with encryption; on the other hand, it sends an instruction to the multimodal perception module to enter the deep monitoring mode.
[0061] Recovery phase (t>240 hours) Thanks to timely early warning and intervention, users' lifestyles began to adjust after T=240 hours. Figure 2 The data shows that sleep duration is gradually returning to the baseline. Figure 1 The LF / HF ratio has fallen below 3.0. Correspondingly, Figure 7 Risk probability in After intervention, the system showed a downward trend (affected by the intervention factor in the algorithm), and finally returned to a safety level below 0.2 after T=300 hours, verifying the effectiveness and closed-loop control capability of the system of the present invention in the entire process from perception to early warning to intervention.
Claims
1. A mental health risk intelligent early warning system based on multimodal data, characterized in that, include: The multimodal perception module is used to collect users' physiological signal data, behavioral trajectory data, and social interaction data; The data processing module is used to perform feature extraction, spatiotemporal alignment, and normalization on the collected data to generate multimodal feature fusion vectors. The dynamic label building module is used to generate weakly supervised pseudo-labels based on heuristic rules, calculate the information entropy of the model's predicted probability distribution, trigger an active query to obtain the user's real feedback labels when the information entropy exceeds the query threshold, and fuse the real feedback labels with the weakly supervised pseudo-labels to generate hybrid training labels. The local training module is used to construct a long short-term memory network model, train the model using the multimodal feature fusion vector and the hybrid training labels, calculate the original gradient of the model parameters, and perform gradient clipping and Gaussian noise injection on the original gradient to generate encrypted noise gradient. The global aggregation module receives encrypted noise gradients, performs federated average aggregation to update global model parameters, and returns the updated parameters to overwrite local model parameters. The interpretive evaluation module is used to calculate the contribution of input features to the model prediction results using the integral gradient algorithm, and generate an explanation of the contribution of key features. The feedback control module is used to dynamically adjust the sampling frequency of the multimodal perception module and the query threshold of the dynamic label construction module based on the risk prediction probability output by the model.
2. The intelligent early warning system for mental health risks based on multimodal data according to claim 1, characterized in that, The feature extraction, spatiotemporal alignment, and normalization processes performed by the data processing module specifically include: The frequency domain features of heart rate variability were extracted from physiological signal data using Fast Fourier Transform. Calculate the location coordinate sequence trajectory entropy and average daily sleep duration of behavioral trajectory data; Statistical analysis of social interaction data, including interaction frequency and voice characteristics; The above features are aligned to a uniform time granularity by means pooling downsampling of high-frequency signals and linear interpolation upsampling of sparse data, and then Z-score normalization is performed to form the multimodal feature fusion vector.
3. The intelligent early warning system for mental health risks based on multimodal data according to claim 1, characterized in that, The logic for generating weakly supervised pseudo-labels by the dynamic label construction module is as follows: Preset discriminant functions for abnormal physiological stress, behavioral withdrawal, and social isolation; When the frequency domain feature of heart rate variability is detected to be higher than the standard deviation of the user's historical mean, and at the same time the sleep duration is lower than the sleep deprivation threshold or the interaction frequency is lower than the social isolation threshold, a weakly supervised pseudo-label indicating high risk is generated. Otherwise, weakly supervised pseudo-labels indicating low risk are generated.
4. The intelligent early warning system for mental health risks based on multimodal data according to claim 1, characterized in that, The tag fusion logic executed by the dynamic tag construction module is as follows: The Shannon entropy is obtained by calculating the sum of the negative logarithms of the predicted probability distributions and using it as the information entropy. When the active query trigger condition is met and the user's real feedback label is obtained, the indicator variable is set to valid, and the real feedback label is given absolute weight in the calculation formula of the mixed training label. When no real user feedback labels are obtained, the indicator variable is set to invalid, and the weakly supervised pseudo-labels are directly used as the mixed training labels.
5. The intelligent early warning system for mental health risks based on multimodal data according to claim 1, characterized in that, The long short-term memory network model in the local training module includes an input mapping layer, an LSTM recurrent layer, and a classification output layer. The LSTM recurrent layer stores long-term dependent information by dynamically updating the cell state through forget gate, input gate and output gate mechanisms, and the classification output layer outputs the current mental health risk prediction probability through the Softmax activation function.
6. The intelligent early warning system for mental health risks based on multimodal data according to claim 1, characterized in that, The specific steps for the local training module to generate encrypted noise gradients include: The L2 norm of the original gradient is obtained by calculating the square root of the sum of the squares of the values of each element in the original gradient vector. The L2 norm of the original gradient is compared with a preset clipping threshold. If the L2 norm of the original gradient is greater than the clipping threshold, the original gradient is scaled according to the ratio of the clipping threshold to the L2 norm of the original gradient to obtain a clipped gradient with a modulus equal to the clipping threshold. If the L2 norm of the original gradient is less than or equal to the clipping threshold, then the original gradient is directly used as the clipping gradient. Based on the preset privacy budget parameters and noise multiplier, a noise vector is generated that follows a multidimensional Gaussian distribution with a mean of zero and a variance related to the pruning threshold. The noise vector is superimposed on the clipping gradient to generate an encrypted noise gradient that satisfies differential privacy constraints.
7. The intelligent early warning system for mental health risks based on multimodal data according to claim 1, characterized in that, The logic for the global aggregation module to perform federated average aggregation is as follows: The proportion of samples from each data source participating in the training to the total number of samples is used as the weight; The weighted summation of all received encrypted noise gradients is used to calculate the update amount of the global model parameters; By using the global learning rate to update the global model parameters, model convergence can be achieved through multi-party collaboration.
8. The intelligent early warning system for mental health risks based on multimodal data according to claim 1, characterized in that, The specific steps for the attribution analysis performed by the interpretive evaluation module include: Set either a zero vector or a historical mean vector as the baseline input; The Riemann approximation method is used to calculate the integral gradient value of the model output gradient relative to the change in input features as it changes from the baseline input along a linear path to the current multimodal feature fusion vector. The feature dimensions with the largest absolute values of the integral gradient are mapped to clinically understandable natural language descriptions, which serve as the explanation for the contribution of the key features.
9. A mental health risk intelligent early warning system based on multimodal data according to claim 1, characterized in that, The feedback control module adjusts the sampling frequency as follows: Set low-risk and high-risk thresholds; When the risk prediction probability is lower than the low-risk threshold, the multimodal perception module is controlled to maintain the baseline sampling frequency. When the risk prediction probability is higher than or equal to the high-risk threshold, the multimodal sensing module is controlled to increase the sampling frequency to a high-frequency state based on the linear growth function or step function of the risk probability.
10. A mental health risk intelligent early warning system based on multimodal data according to claim 9, characterized in that, The feedback control module is also used for: When the risk prediction probability is higher than or equal to the high-risk threshold, a signal is sent to the dynamic tag construction module to lower the query threshold and increase the trigger sensitivity of active query. If the risk prediction probability remains above the high-risk threshold for more than a preset time, the key feature contribution explanation generated by the interpretive assessment module and the risk trend chart will be packaged together to generate an alarm data packet and encrypted and pushed to the external monitoring terminal.