Mental health risk early warning and intervention system based on multi-mode emotion calculation

Abstract

The invention provides a mental health risk early warning and intervention system based on multi-modal emotion calculation, and belongs to the field of mental health analysis, and the system comprises a multi-modal fusion module which is used for obtaining multi-modal data, carrying out the constraint and fusion of the multi-modal data, and generating a fused emotion feature; the risk prediction module is used for calculating a mental health risk probability and an emotional state according to the fused emotional characteristics through a two-stage accumulation risk prediction algorithm, and performing mental health risk early warning based on the emotional state; and the strategy recommendation module is used for extracting emotion track characteristics based on the fused emotion characteristics and the mental health risk probability, and carrying out intervention strategy recommendation to obtain an intervention strategy of the current mental health risk. According to the invention, through the multi-modal fusion module, the risk prediction module and the strategy recommendation module, four modal data of video, audio, text and physiology are processed at the same time, a nonlinear accumulation process of mental health risks is accurately modeled, and risk early warning and intervention capabilities are improved.

Description

Technical Field

[0001] This invention relates to the field of mental health analysis technology, and in particular to a mental health risk early warning and intervention system based on multimodal affective computing. Background Technology

[0002] Mental health assessment technology is an interdisciplinary research field involving multiple disciplines such as psychology, neuroscience, computer science, and biomedical engineering. Traditional mental health assessments mainly rely on psychological testing scales and clinical interviews. These methods are easily influenced by factors such as the test subject's current state, verbal expression ability, and subjective will, leading to issues with the objectivity and accuracy of the assessment results. With the development of physiological signal detection technology, some studies have begun to use physiological data such as heart rate variability and electroencephalogram (EEG) signals for mental state assessment. However, these methods typically focus only on single-modality data and lack comprehensive analysis of multi-dimensional information. Furthermore, most existing technologies can only perform static state assessments, lacking the ability for dynamic risk prediction and proactive intervention recommendations.

[0003] Chinese invention patent CN113017633B discloses an intelligent mental analysis and evaluation method and system based on human characteristic data. It collects human electrocardiogram (ECG) signals, calculates the respiratory fundamental frequency and instantaneous heart rate power spectrum, then calculates mental system characteristic parameters based on the floating frequency band method, performs heart rate variability analysis, and obtains periodic mental state scores through data mining algorithms. Finally, it matches human health data with corresponding scores according to preset rules to generate personalized evaluations. However, it mainly relies on a single ECG signal and has relatively poor real-time performance, which may affect the system's reliability under unstable network connections. Summary of the Invention

[0004] In view of this, the present invention proposes a mental health risk early warning and intervention system based on multimodal emotion computing, which can simultaneously process four modalities of data: video, audio, text and physiological data. Through the risk prediction module, it accurately describes the nonlinear accumulation process of mental health risks, thereby improving the risk early warning and intervention capabilities.

[0005] The technical solution of this invention is implemented as follows: This invention provides a mental health risk early warning and intervention system based on multimodal emotion computing, comprising:

[0006] The multimodal fusion module is used to acquire multimodal data, and uses an emotion fusion network to constrain and fuse the multimodal data to generate fused emotion features;

[0007] The risk prediction module is used to calculate the probability of mental health risk based on the fused emotional features using a two-stage cumulative risk prediction algorithm, determine the emotional state based on the probability of mental health risk, and provide a mental health risk warning based on the emotional state.

[0008] The strategy recommendation module is used to extract emotional trajectory features based on fused emotional features and mental health risk probabilities, and to recommend intervention strategies based on the emotional trajectory features to obtain intervention strategies for the current mental health risks.

[0009] Based on the above technical solutions, preferably, the multimodal data includes video data, audio data, text data, and physiological data, wherein,

[0010] The video data includes facial expression feature data, eye tracking data, posture feature data, and behavioral action sequence data;

[0011] The audio data includes speech emotion feature data, intonation change data, speech rate rhythm data, and acoustic spectrum feature data;

[0012] The text data includes sentiment lexical feature data, language expression style data, syntactic structure feature data, and semantic sentiment tendency data;

[0013] The physiological data includes heart rate variability data, skin conductance data, brain electrical activity data, and respiratory pattern data.

[0014] Based on the above technical solutions, preferably, the emotion fusion network includes a standardization layer, a feature extraction layer, a mapping layer, a bilinear pooling interaction layer, a dynamic weight calculation layer, and a fusion layer, wherein,

[0015] The standardization layer is used to standardize the multimodal data to obtain standardized multimodal data.

[0016] The feature extraction layer extracts features from the standardized multimodal data to obtain video features, audio features, text features, and physiological features;

[0017] The mapping layer maps the video features, audio features, text features, and physiological features to a vector space of a unified dimension, thereby obtaining constrained modal features.

[0018] The bilinear pooling interaction layer calculates the interaction relationship between each modal feature after constraint, and determines the cross-modal interaction feature based on the interaction relationship;

[0019] The dynamic weight calculation layer calculates the dynamic fusion weight of the multimodal data at the current moment;

[0020] The fusion layer fuses multimodal data based on dynamic fusion weights and cross-modal interaction features to generate fused sentiment features.

[0021] Based on the above technical solutions, preferably, the calculation formula for the fused emotional features is as follows:

[0022] M(t)=∑ i∈v,a,txt,p α i (t)·E i (t)+β(t)·C(t);

[0023] β(t)=σ(W β ×[∑ i∈v,a,txt,p E i (t)]+b β );

[0024] C(t) = W c ×B(t)+b c ;

[0025] Where M(t) represents the fused sentiment feature at time t, i represents the modality index, v represents the video feature identifier, a represents the audio feature identifier, txt represents the text feature identifier, p represents the physiological feature identifier, and α represents the fused sentiment feature at time t. i (t) represents the dynamic fusion weight of mode i at time t, E i β(t) represents the modal features of modality i at time t after semantic mapping, β(t) represents the modulation weights of cross-modal interaction features at time t, C(t) represents the cross-modal interaction features at time t, σ represents the sigmoid activation function, and W β b represents the parameter matrix for calculating the interaction weights. β W represents the bias vector for calculating interaction weights. c Let B(t) represent the interaction feature mapping matrix, and let B(t) represent the cross-modal interaction feature vector calculated through bilinear pooling. c This represents the bias vector of the interactive feature mapping matrix.

[0026] Based on the above technical solution, preferably, the two-stage cumulative risk prediction algorithm includes two branch calculation strategies, namely a first branch calculation strategy and a second branch calculation strategy, wherein,

[0027] The fused emotional features are extracted and the emotional load is accumulated to obtain the accumulated emotional load;

[0028] The accumulated emotional load is compared with a preset risk activation threshold:

[0029] If the accumulated emotional load is not greater than the preset risk activation threshold, the first branch calculation strategy is adopted to obtain the probability of mental health risk.

[0030] If the accumulated emotional load exceeds the preset risk activation threshold, the second branch calculation strategy is adopted to obtain the probability of mental health risk.

[0031] Based on the above technical solutions, preferably, the extraction of the fused emotional features and the accumulation of emotional load specifically include:

[0032] The fused emotional features are extracted using a bipolar activation function to obtain the intensity of the emotional state at the current moment.

[0033] The emotional load value is obtained by calculating the portion of the emotional state intensity that exceeds a preset critical threshold using a maximum value function.

[0034] The cumulative emotional load is obtained by modulating the emotional load value using an adaptive decay recursive update mechanism.

[0035] Based on the above technical solutions, the preferred logic of the two-stage cumulative risk prediction algorithm is as follows:

[0036]

[0037] Wherein, formula (1) represents the first branch calculation strategy; formula (2) represents the second branch calculation strategy; and P risk (t+Δt) represents the probability of mental health risk at time t+Δt, where Δt represents the prediction time interval, and P base θ represents the basic risk probability. danger L represents the characteristic parameter of a dangerous state. stage1 (t) represents the cumulative emotional load at time t, θ critical Indicates the preset risk activation threshold, L stage2 (t) represents the risk trigger value at time t;

[0038] When the accumulated emotional load exceeds the preset risk activation threshold, the emotional load risk value is obtained by comparing the accumulated emotional load with the preset risk activation threshold, and the emotional load risk value is amplified by a nonlinear amplification mechanism to obtain the risk increment value.

[0039] The risk trigger value is obtained by linearly superimposing the accumulated emotional load and the risk increment value. The calculation formula is as follows:

[0040] L stage2 (t)=L stage1 (t)+φ·tanh(κ·(L stage1 (t)-θ critical ));

[0041] Where φ represents the nonlinear amplification factor, tanh(·) represents the hyperbolic tangent function, and k represents the activation sensitivity parameter.

[0042] Based on the above technical solutions, preferably, the steps for determining the hazardous state characteristic parameters are as follows:

[0043] Preset danger state indicators are extracted from the fused emotional features, including the amplitude of emotional fluctuations, the intensity of negative emotions, and the duration of the emotional state;

[0044] Based on the weighted combination calculation of the amplitude of the emotional fluctuation, the intensity of the negative emotion, and the duration of the emotional state, the characteristic parameters of the dangerous state are obtained.

[0045] More preferably, the execution steps of the strategy recommendation module include:

[0046] A temporal sequence of emotional trajectories is constructed based on the emotional state and the probability of mental health risk.

[0047] Local features, global trend features, and frequency domain pattern features of the emotional trajectory are extracted from the emotional trajectory time series.

[0048] The local features, global trend features, and frequency domain pattern features are fused using an attention weighting mechanism to obtain comprehensive trajectory features;

[0049] The comprehensive trajectory features are matched with the historical trajectory database in multiple dimensions to obtain the emotional trajectory matching result; wherein the historical trajectory database includes historical emotional trajectories and corresponding intervention strategies.

[0050] Intervention strategies are extracted from the historical trajectory database based on the emotional trajectory matching results.

[0051] Based on the above technical solutions, preferably, the multi-dimensional matching specifically includes:

[0052] Calculate the feature similarity, cosine similarity, and frequency domain similarity between the comprehensive trajectory features and the historical emotional trajectory, respectively;

[0053] The emotional trajectory matching results are determined based on feature similarity, cosine similarity, and frequency domain similarity.

[0054] The mental health risk early warning and intervention system based on multimodal affective computing of the present invention has the following advantages over the prior art:

[0055] (1) The multimodal fusion module, risk prediction module and strategy recommendation module simultaneously process video, audio, text and physiological data. The risk prediction module accurately describes the nonlinear accumulation process of mental health risks, thereby improving the risk warning and intervention capabilities.

[0056] (2) By mapping different modal features to a unified dimensional vector space and using dynamic weight fusion, the dynamic weight calculation layer can adaptively adjust the fusion weight according to the data quality of each modality at the current time. The introduction of cross-modal interaction features enhances the information complementarity between modalities and significantly improves the expression quality and discrimination ability of fused emotional features.

[0057] (3) The two-stage cumulative risk prediction algorithm accurately reflects the stage characteristics and nonlinear change law of mental health risk formation, avoids the limitations of a single linear model, and improves the pertinence of prediction and risk assessment ability. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a block diagram of a mental health risk early warning and intervention system based on multimodal emotion computing according to the present invention;

[0060] Figure 2 This is a schematic diagram of a multimodal fusion module of a mental health risk early warning and intervention system based on multimodal emotion computing according to the present invention;

[0061] Figure 3 This is a schematic diagram of a two-stage cumulative risk prediction algorithm for a mental health risk early warning and intervention system based on multimodal emotion computing, according to the present invention. Detailed Implementation

[0062] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0063] like Figure 1 As shown, this invention provides a mental health risk early warning and intervention system based on multimodal emotion computing, comprising:

[0064] The multimodal fusion module is used to acquire multimodal data, and uses an emotion fusion network to constrain and fuse the multimodal data to generate fused emotion features;

[0065] The risk prediction module is used to calculate the probability of mental health risk based on the fused emotional features using a two-stage cumulative risk prediction algorithm, determine the emotional state based on the probability of mental health risk, and provide a mental health risk warning based on the emotional state.

[0066] The strategy recommendation module is used to extract emotional trajectory features based on fused emotional features and mental health risk probabilities, and to recommend intervention strategies based on the emotional trajectory features to obtain intervention strategies for the current mental health risks.

[0067] This invention achieves the process from acquiring multi-source heterogeneous data to accurate risk prediction and personalized intervention strategy recommendation through a multimodal fusion module, a risk prediction module, and a strategy recommendation module. It can simultaneously process four modalities of data: video, audio, text, and physiological data. By fusing emotional semantic space, it avoids the dimensionality curse problem of traditional feature splicing methods. The risk prediction module accurately describes the nonlinear accumulation process of mental health risks. The strategy recommendation mechanism based on emotional trajectory matching enables the effective reuse of historical successful cases, thereby improving risk warning and intervention capabilities.

[0068] Specifically, the multimodal data includes video data, audio data, text data, and physiological data, among which,

[0069] The video data includes facial expression feature data, eye tracking data, posture feature data, and behavioral action sequence data;

[0070] The audio data includes speech emotion feature data, intonation change data, speech rate rhythm data, and acoustic spectrum feature data;

[0071] The text data includes sentiment lexical feature data, language expression style data, syntactic structure feature data, and semantic sentiment tendency data;

[0072] The physiological data includes heart rate variability data, skin conductance data, brain electrical activity data, and respiratory pattern data.

[0073] Understandably, video data can reflect an individual's external emotional expression and changes in behavioral patterns, audio data can capture the emotional information and psychological state changes contained in an individual's voice, text data is extracted from an individual's language expression through natural language processing technology, and can reflect an individual's cognitive state and emotional tendencies, while physiological data is acquired through wearable devices and physiological sensors, and can directly reflect an individual's physiological arousal level and autonomic nervous system activity.

[0074] like Figure 2 As shown, in one embodiment of the present invention, the emotion fusion network includes a normalization layer, a feature extraction layer, a mapping layer, a bilinear pooling interaction layer, a dynamic weight calculation layer, and a fusion layer, wherein...

[0075] The standardization layer is used to standardize the multimodal data to obtain standardized multimodal data.

[0076] The feature extraction layer extracts features from the standardized multimodal data to obtain video features, audio features, text features, and physiological features;

[0077] The mapping layer maps the video features, audio features, text features, and physiological features to a vector space of a unified dimension, thereby obtaining constrained modal features.

[0078] The bilinear pooling interaction layer calculates the interaction relationship between each modal feature after constraint, and determines the cross-modal interaction feature based on the interaction relationship;

[0079] The dynamic weight calculation layer calculates the dynamic fusion weight of the multimodal data at the current moment;

[0080] The fusion layer fuses multimodal data based on dynamic fusion weights and cross-modal interaction features to generate fused sentiment features.

[0081] Specifically, the formula for calculating the fused emotional features is as follows:

[0082] M(t)=∑ i∈v,a,txt,p α i (t)·E i (t)+β(t)·C(t);

[0083] β(t)=σ(W β ×[∑ i∈v,a,txt,p E i (t)]+b β );

[0084] C(t) = W c ×B(t)+b c ;

[0085] Where M(t) represents the fused sentiment feature at time t, i represents the modality index, v represents the video feature identifier, a represents the audio feature identifier, txt represents the text feature identifier, p represents the physiological feature identifier, and α represents the fused sentiment feature at time t. i (t) represents the dynamic fusion weight of mode i at time t, E i β(t) represents the modal features of modality i at time t after semantic mapping, β(t) represents the modulation weights of cross-modal interaction features at time t, C(t) represents the cross-modal interaction features at time t, σ represents the sigmoid activation function, and W β b represents the parameter matrix for calculating the interaction weights. β W represents the bias vector for calculating interaction weights. cLet B(t) represent the interaction feature mapping matrix, and let B(t) represent the cross-modal interaction feature vector calculated through bilinear pooling. c This represents the bias vector of the interactive feature mapping matrix.

[0086] Understandably, different modalities of data differ in terms of expressive power and reliability. A dynamic weighting mechanism can adaptively adjust the fusion weights based on the quality of each modality of data at the current moment, avoiding the limitations of fixed weight allocation. This is achieved through dynamic weight α. i (t) Weighted summation of each modal feature is performed, and cross-modal interaction features are added to achieve multimodal data fusion.

[0087] By mapping different modal features to a vector space of the same dimension and using dynamic weight fusion, the technical problems of semantic misalignment and fixed weight allocation between modalities in traditional multimodal fusion are solved. The mapping layer ensures the comparability of different modalities in the same semantic space, the dynamic weight calculation layer can adaptively adjust the fusion weights according to the data quality of each modality at the current time, and the introduction of cross-modal interaction features enhances the information complementarity between modalities, thereby significantly improving the expression quality and discrimination ability of fused sentiment features.

[0088] In one embodiment of the present invention, the normalization layer uses a zero-mean unit variance normalization method to process pixel values ​​and feature vectors for video data; a power spectral density-based normalization method to process acoustic features for audio data; a TF-IDF weight and word vector normalization method to process text features for text data; and a Z-score normalization method based on individual baselines to process physiological signals for physiological data. Through normalization processing, the dimensional differences and numerical biases between different modal data can be effectively eliminated.

[0089] In one embodiment of the present invention, the feature extraction layer uses an improved 3D convolutional neural network to extract spatiotemporal features from video data. This network can simultaneously capture the spatial distribution and temporal evolution information of facial expressions. For audio data, a hybrid network combining convolutional and recurrent structures is used to extract the spectral-temporal features of the audio. This network can effectively process the frequency domain and temporal domain features of speech signals. For text data, a deep learning language model is used to extract the semantic features of the text. This model can understand the semantic relationships of the context. For physiological data, a one-dimensional convolutional network combined with a long short-term memory network is used to extract multi-scale features of physiological signals. This network can simultaneously process the instantaneous fluctuations and long-term trends of physiological signals.

[0090] In one embodiment of the present invention, the mapping layer maps features from different modalities to a unified 128-dimensional sentiment semantic vector space, and the calculation formula is as follows:

[0091] E v (t)=tanh(W v·V(t)+b v )

[0092] E a (t)=tanh(W a ·A(t)+b a )

[0093] E txt (t)=tanh(W txt ·T(t)+b txt )

[0094] E p (t)=tanh(W p ·P(t)+b p )

[0095] Among them, E v (t) represents the emotional semantic representation vector of the video data at time t, tanh(·) represents the hyperbolic tangent function, and W v Let V(t) represent the learnable semantic mapping weight matrix of the video data, and let b represent the original feature vector obtained after the video data is processed by the feature extraction network at time t. v E represents the bias vector for semantic mapping of video data. a W(t) represents the emotional semantic representation vector of the audio data at time t. a Let A(t) represent the learnable semantic mapping weight matrix of the audio data, and let b represent the original feature vector obtained after the audio data is processed by the feature extraction network at time t. a E represents the bias vector that maps the semantics of audio data. txt W(t) represents the sentiment semantic representation vector of the text data at time t. txt Let T(t) represent the learnable semantic mapping weight matrix of the text data, and let T(t) represent the original feature vector of the text data at time t after being extracted by the natural language processing network. txt E represents the bias vector that maps the semantics of text data. p W(t) represents the emotional semantic representation vector of physiological data at time t. p Let P(t) represent the learnable semantic mapping weight matrix of the physiological data, and let b represent the original feature vector obtained by the signal processing network after the physiological data is extracted at time t. p The bias vector represents the semantic mapping of physiological data.

[0096] In one embodiment of the present invention, the dynamic weight calculation layer calculates the importance score of each modality semantic vector through an attention mechanism, and calculates the dynamic weight of each modality data based on the importance score. The calculation formula is as follows:

[0097]

[0098] s i (t)=W s T ×E i (t)+b s

[0099] Among them, s i (t) represents the importance score of mode i at time t, s j (t) represents the importance score of mode j at time t, where j represents the mode index other than i.

[0100] Understandably, by calculating the importance score of each modality and then normalizing it using the softmax function, a dynamic weight in probabilistic form is obtained, representing the importance score s. i (t) is calculated through a linear transformation, reflecting the relative importance of mode i at the current moment.

[0101] like Figure 3 As shown, in one embodiment of the present invention, the two-stage cumulative risk prediction algorithm includes two branch calculation strategies, namely a first branch calculation strategy and a second branch calculation strategy, wherein,

[0102] The fused emotional features are extracted and the emotional load is accumulated to obtain the accumulated emotional load;

[0103] The accumulated emotional load is compared with a preset risk activation threshold:

[0104] If the accumulated emotional load is not greater than the preset risk activation threshold, the first branch calculation strategy is adopted to obtain the probability of mental health risk.

[0105] If the accumulated emotional load exceeds the preset risk activation threshold, the second branch calculation strategy is adopted to obtain the probability of mental health risk.

[0106] Specifically, the extraction of the fused emotional features and the accumulation of emotional load include:

[0107] The fused emotional features are extracted using a bipolar activation function to obtain the emotional state intensity at the current moment. The calculation formula is as follows:

[0108]

[0109] Among them, Stress instant (t) represents the intensity of the instantaneous emotional state at time t, σ stress (x) is an improved bipolar sigmoid function that can handle both positive and negative stress conditions, w stress To extract weight vectors for stress response,

[0110] Let M(t) represent the transpose of the stress extraction weight vector, and let M(t) represent the fused sentiment features at time t. stress For bias terms;

[0111] The emotional load value is obtained by calculating the portion of the emotional state intensity that exceeds a preset critical threshold using a maximum value function.

[0112] The emotional load value is modulated using an adaptive decay recursive update mechanism to obtain the cumulative emotional load, calculated as follows:

[0113] L stage1 (t)=η(t)·L stage1 (t-1)+λ·max(0,Stress instant (t)-θ tolerance )

[0114] η(t)=η base +(1-η base )·exp(-ρ·|Stress instant (t)|)

[0115] Among them, L stage1 (t) represents the cumulative emotional load at time t, η(t) represents the adaptive decay factor at time t, and L stage1 (t-1) represents the cumulative emotional load at the previous moment, λ represents the cumulative coefficient, max(·) represents the maximum value function, and θ tolerance Indicates the preset critical threshold, η base ρ represents the basic attenuation factor, and ρ represents the stress response sensitivity parameter.

[0116] Understandably, the intensity of the instantaneous emotional state at time t is Stress. instant The value range of (t) is [-1, 1], with negative values ​​representing positive emotional states and positive values ​​representing negative stress states. The absolute value reflects the intensity of the emotion. The bipolar activation function can distinguish between positive and negative emotional states, while ensuring the boundedness and smoothness of the output through the tanh function. The cumulative coefficient λ controls the contribution ratio of the new load to the overall cumulative load, and is usually set between 0.1 and 0.3. The basic attenuation factor η base Typically set between 0.85 and 0.95, it reflects an individual's baseline recovery capacity under normal conditions. The stress response sensitivity parameter ρ controls the degree of response of the attenuation factor to the current stress level, and is typically set between 1.0 and 3.0. When the instantaneous stress intensity is high, exp(-ρ·|Stress instant (t)|) decreases, making η(t) approach η baseThis indicates that an individual's natural recovery ability is limited under high stress; when the instantaneous stress intensity is low, exp(-ρ·|Stress) instant The fact that η(t) approaches 1 makes η(t) approach 1, indicating that individuals have a strong ability to recover under low pressure.

[0117] In one embodiment of the present invention, the logic of the two-stage cumulative risk prediction algorithm is as follows:

[0118]

[0119] Wherein, formula (1) represents the first branch calculation strategy; formula (2) represents the second branch calculation strategy; and P risk (t+Δt) represents the probability of mental health risk at time t+Δt, where Δt represents the prediction time interval, and P base θ represents the basic risk probability. danger L represents the characteristic parameter of a dangerous state. stage1 (t) represents the cumulative emotional load at time t, θ critical Indicates the preset risk activation threshold, L stage2 (t) represents the risk trigger value at time t;

[0120] When the accumulated emotional load exceeds the preset risk activation threshold, the emotional load risk value is obtained by comparing the accumulated emotional load with the preset risk activation threshold, and the emotional load risk value is amplified by a nonlinear amplification mechanism to obtain the risk increment value.

[0121] The risk trigger value is obtained by linearly superimposing the accumulated emotional load and the risk increment value. The calculation formula is as follows:

[0122] L stage2 (t)=L stage1 (t)+φ·tanh(k·(L stage1 (t)-θ critical ));

[0123] Where φ represents the nonlinear amplification factor, tanh(·) represents the hyperbolic tangent function, and k represents the activation sensitivity parameter.

[0124] Understandably, the nonlinear amplification factor φ controls the amplification intensity during the risk excitation phase. Specifically, the formula for calculating the nonlinear amplification factor is:

[0125]

[0126] Where, φ base δ represents the fundamental amplification factor, δ represents the periodic modulation intensity coefficient, sin(·) represents the sine function, and ω represents the circadian rhythm frequency. This indicates the phase shift, and t represents the time.

[0127] Understandably, the biological rhythm frequency ω is usually set to 2π / 24 (in hours), corresponding to the 24-hour circadian rhythm cycle; phase shift It is used to adjust the phase of an individual's biological rhythm, and is usually personalized based on the individual's daily routine. The value range is [-1, 1]. Positive values ​​usually correspond to the individual's physiologically active period (such as daytime), and negative values ​​correspond to the individual's physiologically resting period (such as nighttime). This periodic modulation allows the intensity of risk arousal to reflect the diurnal variation pattern of human mental state.

[0128] The two-stage cumulative risk prediction algorithm accurately reflects the stage-specific characteristics and non-linear change patterns of mental health risk formation, avoiding the limitations of a single linear model and improving the pertinence of prediction and risk assessment capabilities.

[0129] Furthermore, the steps for determining the hazardous state characteristic parameters are as follows:

[0130] Preset danger state indicators are extracted from the fused emotional features, including the amplitude of emotional fluctuations, the intensity of negative emotions, and the duration of the emotional state;

[0131] Based on the weighted combination calculation of the amplitude of the emotional fluctuation, the intensity of the negative emotion, and the duration of the emotional state, the characteristic parameters of the dangerous state are obtained.

[0132] In one embodiment of the present invention, the following grading standard is used to determine the state of mental health risk:

[0133] When P risk (t+Δt)<0.3 indicates a low-risk state; routine monitoring is required without special intervention.

[0134] When 0.3≤P risk (t+Δt)<0.6 indicates a medium-risk situation, triggering a yellow alert and recommending preventative intervention.

[0135] When 0.6≤P risk (t+Δt)<0.8 indicates a high-risk situation, triggering an orange alert and recommending immediate professional intervention.

[0136] When P risk If (t+Δt)≥0.8, it is determined to be an extremely high-risk state, a red alert is issued, and emergency medical intervention is recommended.

[0137] In one embodiment of the present invention, the execution steps of the strategy recommendation module include:

[0138] A temporal sequence of emotional trajectories is constructed based on the emotional state and the probability of mental health risk.

[0139] Local features, global trend features, and frequency domain pattern features of the emotional trajectory are extracted from the emotional trajectory time series.

[0140] The local features, global trend features, and frequency domain pattern features are fused using an attention weighting mechanism to obtain comprehensive trajectory features;

[0141] The comprehensive trajectory features are matched with the historical trajectory database in multiple dimensions to obtain the emotional trajectory matching result; wherein the historical trajectory database includes historical emotional trajectories and corresponding intervention strategies.

[0142] Intervention strategies are extracted from the historical trajectory database based on the emotional trajectory matching results.

[0143] By constructing a time-series sequence of emotional trajectories and extracting local features, global trend features, and frequency domain pattern features, this approach comprehensively captures multi-level information about the individual's emotional evolution process. It avoids the inaccuracy of information represented by single-scale features and overcomes the bias and limitations of single similarity measures through multi-dimensional matching, thereby improving the accuracy and reliability of intervention strategy recommendations.

[0144] Specifically, the multi-dimensional matching includes:

[0145] Calculate the feature similarity, cosine similarity, and frequency domain similarity between the comprehensive trajectory features and the historical emotional trajectory, respectively;

[0146] The emotional trajectory matching results are determined based on feature similarity, cosine similarity, and frequency domain similarity.

[0147] In one embodiment of the present invention, local features are achieved using a one-dimensional convolutional neural network. By sliding convolutional kernels of different sizes on the trajectory sequence, local patterns at different time scales are captured. The one-dimensional convolutional neural network contains multiple convolutional layers and pooling layers, which can automatically learn local patterns and features in the trajectory. By applying convolutional operations at all time positions, local features of the entire trajectory sequence are extracted. These features can capture short-term change patterns of emotional states and local anomalies.

[0148] In one embodiment of the present invention, global trend feature extraction employs a gated recurrent unit network to capture the long-term evolution trend and overall development direction of emotional trajectories. The gating mechanism can effectively process long sequence data and automatically learn which historical information is more important for predicting future trends. The internal update mechanism of the gated recurrent unit includes the calculation of the reset gate, the update gate, and the candidate state, which can effectively solve the gradient vanishing problem in traditional recurrent neural networks when processing long sequences.

[0149] In one embodiment of the present invention, frequency domain pattern features are identified by frequency domain analysis technology to identify the periodic patterns of emotional changes. The time domain trajectory signal is converted into a frequency domain representation by Fourier transform to identify the main frequency components and periodic patterns.

[0150] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A mental health risk early warning and intervention system based on multimodal affective computing, characterized in that: include: The multimodal fusion module is used to acquire multimodal data, and uses an emotion fusion network to constrain and fuse the multimodal data to generate fused emotion features; The risk prediction module is used to calculate the probability of mental health risk based on the fused emotional features using a two-stage cumulative risk prediction algorithm, determine the emotional state based on the probability of mental health risk, and provide a mental health risk warning based on the emotional state. The strategy recommendation module is used to extract emotional trajectory features based on fused emotional features and mental health risk probabilities, and to recommend intervention strategies based on the emotional trajectory features to obtain intervention strategies for the current mental health risks.

2. The mental health risk early warning and intervention system based on multimodal affective computing as described in claim 1, characterized in that: The multimodal data includes video data, audio data, text data, and physiological data, among which... The video data includes facial expression feature data, eye tracking data, posture feature data, and behavioral action sequence data; The audio data includes speech emotion feature data, intonation change data, speech rate rhythm data, and acoustic spectrum feature data; The text data includes sentiment lexical feature data, language expression style data, syntactic structure feature data, and semantic sentiment tendency data; The physiological data includes heart rate variability data, skin conductance data, brain electrical activity data, and respiratory pattern data.

3. A mental health risk early warning and intervention system based on multimodal affective computing as described in claim 2, characterized in that: The emotion fusion network comprises a normalization layer, a feature extraction layer, a mapping layer, a bilinear pooling interaction layer, a dynamic weight calculation layer, and a fusion layer, wherein... The standardization layer is used to standardize the multimodal data to obtain standardized multimodal data. The feature extraction layer extracts features from the standardized multimodal data to obtain video features, audio features, text features, and physiological features; The mapping layer maps the video features, audio features, text features, and physiological features to a vector space of a unified dimension, thereby obtaining constrained modal features. The bilinear pooling interaction layer calculates the interaction relationship between each modal feature after constraint, and determines the cross-modal interaction feature based on the interaction relationship; The dynamic weight calculation layer calculates the dynamic fusion weight of the multimodal data at the current moment; The fusion layer fuses multimodal data based on dynamic fusion weights and cross-modal interaction features to generate fused sentiment features.

4. A mental health risk early warning and intervention system based on multimodal affective computing as described in claim 3, characterized in that: The formula for calculating the fused emotional features is as follows: M(t)=∑ i∈v,a,txt,p α i (t)·E i (t)+β(t)·C(t); β(t)=σ(W β ×[∑ i∈v,a,txt,p E i (t)]+b β ); C(t)=W c ×B(t)+b c ; Where M(t) represents the fused sentiment feature at time t, i represents the modality index, v represents the video feature identifier, a represents the audio feature identifier, txt represents the text feature identifier, p represents the physiological feature identifier, and α represents the fused sentiment feature at time t. i (t) represents the dynamic fusion weight of mode i at time t, E i β(t) represents the modal features of modality i at time t after semantic mapping, β(t) represents the modulation weights of cross-modal interaction features at time t, C(t) represents the cross-modal interaction features at time t, σ represents the sigmoid activation function, and W β b represents the parameter matrix for calculating the interaction weights. β W represents the bias vector for calculating interaction weights. c Let B(t) represent the interaction feature mapping matrix, and let B(t) represent the cross-modal interaction feature vector calculated through bilinear pooling. c This represents the bias vector of the interactive feature mapping matrix.

5. A mental health risk early warning and intervention system based on multimodal affective computing as described in claim 1, characterized in that: The two-stage cumulative risk prediction algorithm includes two branch calculation strategies: a first branch calculation strategy and a second branch calculation strategy. The fused emotional features are extracted and the emotional load is accumulated to obtain the accumulated emotional load; The accumulated emotional load is compared with a preset risk activation threshold: If the accumulated emotional load is not greater than the preset risk activation threshold, the first branch calculation strategy is adopted to obtain the probability of mental health risk. If the accumulated emotional load exceeds the preset risk activation threshold, the second branch calculation strategy is adopted to obtain the probability of mental health risk.

6. A mental health risk early warning and intervention system based on multimodal affective computing as described in claim 5, characterized in that, The extraction of the fused emotional features and the accumulation of emotional load specifically include: The fused emotional features are extracted using a bipolar activation function to obtain the intensity of the emotional state at the current moment. The emotional load value is obtained by calculating the portion of the emotional state intensity that exceeds a preset critical threshold using a maximum value function. The cumulative emotional load is obtained by modulating the emotional load value using an adaptive decay recursive update mechanism.

7. A mental health risk early warning and intervention system based on multimodal affective computing as described in claim 5, characterized in that: The logic of the two-stage cumulative risk prediction algorithm is as follows: Wherein, formula (1) represents the first branch calculation strategy; formula (2) represents the second branch calculation strategy; and P risk (t+Δt) represents the probability of mental health risk at time t+Δt, where Δt represents the prediction time interval, and P base θ represents the basic risk probability. danger L represents the characteristic parameter of a hazardous state. stage1 (t) represents the cumulative emotional load at time t, θ critical Indicates the preset risk activation threshold, L stage2 (t) represents the risk trigger value at time t; When the accumulated emotional load exceeds the preset risk activation threshold, the emotional load risk value is obtained by comparing the accumulated emotional load with the preset risk activation threshold, and the emotional load risk value is amplified by a nonlinear amplification mechanism to obtain the risk increment value. The risk trigger value is obtained by linearly superimposing the accumulated emotional load and the risk increment value. The calculation formula is as follows: L stage2 (t)=L stage1 (t)+φ·tanh(κ·(L stage1 (t)-θ critical )); Where φ represents the nonlinear amplification factor, tanh(·) represents the hyperbolic tangent function, and κ represents the activation sensitivity parameter.

8. A mental health risk early warning and intervention system based on multimodal affective computing as described in claim 7, characterized in that: The steps for determining the characteristic parameters of the hazardous state are as follows: Preset danger state indicators are extracted from the fused emotional features, including the amplitude of emotional fluctuations, the intensity of negative emotions, and the duration of the emotional state; Based on the weighted combination calculation of the amplitude of the emotional fluctuation, the intensity of the negative emotion, and the duration of the emotional state, the characteristic parameters of the dangerous state are obtained.

9. A mental health risk early warning and intervention system based on multimodal affective computing as described in claim 1, characterized in that: The execution steps of the strategy recommendation module include: A temporal sequence of emotional trajectories is constructed based on the emotional state and the probability of mental health risk. Local features, global trend features, and frequency domain pattern features of the emotional trajectory are extracted from the emotional trajectory time series. The local features, global trend features, and frequency domain pattern features are fused using an attention weighting mechanism to obtain comprehensive trajectory features; The comprehensive trajectory features are matched with the historical trajectory database in multiple dimensions to obtain the emotional trajectory matching result; wherein the historical trajectory database includes historical emotional trajectories and corresponding intervention strategies. Intervention strategies are extracted from the historical trajectory database based on the emotional trajectory matching results.

10. A mental health risk early warning and intervention system based on multimodal affective computing as described in claim 9, characterized in that: The multi-dimensional matching specifically includes: Calculate the feature similarity, cosine similarity, and frequency domain similarity between the comprehensive trajectory features and the historical emotional trajectory, respectively; The emotional trajectory matching results are determined based on feature similarity, cosine similarity, and frequency domain similarity.

Images