A mental health warning method and system for mental health education

Abstract

The application discloses a mental health early warning method and system for mental health education, relates to the technical field of mental health monitoring, and comprises the following steps: collecting multi-modal data and performing pretreatment, extracting feature vectors of the pretreated multi-modal data for fusion, using delay embedding to construct phase space trajectory points to calculate trajectory complexity, using weighted summation to calculate a mental health score and early warning, constructing a visual interface to display the mental health score in real time, and storing multi-modal data generated by collection and analysis.

Description

Technical Field

[0001] This invention relates to the field of mental health monitoring technology, and in particular to a mental health early warning method and system for mental health education. Background Technology

[0002] With increasing societal attention to mental health issues, research on mental health education and early warning methods has gradually become a hot topic. Traditional mental health monitoring mainly relies on questionnaires and self-reports. Although these methods can reflect mental health status to some extent, they are highly subjective and cannot dynamically capture an individual's emotional changes in different situations, thus affecting the accuracy and real-time nature of the assessment results. With the rapid development of artificial intelligence and multimodal data processing technologies, mental health monitoring systems based on emotion recognition have gradually entered the research field. Multimodal data fusion technology, by combining physiological behavioral features such as voice and facial expressions, can more comprehensively depict an individual's emotional state. However, common multimodal emotion recognition technologies still have some shortcomings, such as noise interference affecting recognition accuracy, poor feature fusion effects, and a lack of effective modeling of data temporal features, which limit their application in mental health early warning.

[0003] The application of multimodal data in mental health early warning still has some obvious shortcomings. Existing technologies often rely on a single model or simple feature averaging methods in the process of feature extraction and fusion, lacking effective control over data redundancy and noise, which leads to biases in emotion recognition and mental health scoring. Existing mental health early warning systems mostly remain at the static analysis stage when analyzing data, failing to make full use of the dynamic emotional fluctuation characteristics contained in time series data, and failing to effectively capture the fluctuation patterns of individual psychological states, resulting in low accuracy in identifying high-risk emotional states. Summary of the Invention

[0004] In view of the problems existing in the current mental health early warning methods and systems for mental health education, this invention is proposed.

[0005] Therefore, the problem that this invention aims to solve is that existing technologies, in the process of feature extraction and fusion, usually rely on a single model or a simple feature averaging method, lacking effective control over data redundancy and noise, which leads to deviations in emotion recognition and mental health scoring. Existing mental health early warning systems mostly remain at the static analysis stage when analyzing data, failing to fully utilize the dynamic emotional fluctuation characteristics contained in time series data, and are unable to effectively capture the fluctuation patterns of individual psychological states, resulting in low accuracy in identifying high-risk emotional states.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a mental health early warning method for mental health education, comprising: collecting multimodal data and preprocessing it; extracting feature vectors from the preprocessed multimodal data and fusing them; using delayed embedding to construct phase space trajectory points to calculate trajectory complexity; using weighted summation to calculate mental health scores and early warnings; constructing a visual interface to display mental health scores in real time; and storing the collected and analyzed multimodal data.

[0007] As a preferred embodiment of the mental health early warning method for mental health education described in this invention, the step of collecting and preprocessing multimodal data refers to collecting and preprocessing the user's multimodal data using a high-definition camera and microphone during the mental health education process.

[0008] The multimodal data includes voice and facial expression data;

[0009] The preprocessing includes adding timestamps to the collected multimodal data, performing time synchronization correction on the multimodal data using linear interpolation, removing noise from the speech data using a bandpass filter, standardizing the denoised speech data, removing noise from the facial expression data using Gaussian filtering, sharpening the facial expression data using the Laplacian operator, and aligning the facial expression data using affine transformation.

[0010] As a preferred embodiment of the mental health early warning method for mental health education described in this invention, the step of extracting and fusing the feature vectors of preprocessed multimodal data refers to collecting historical multimodal data and performing preprocessing and feature vector extraction to generate a training set;

[0011] The pre-trained FER+ model was used to classify emotions in historical speech data and assign corresponding emotion labels, and the pre-trained OpenSMILE model was used to classify emotions in historical facial expression data and assign corresponding emotion labels.

[0012] The pre-trained OpenAI Whisper model was used to extract speech feature vectors from historical speech data, and the pre-trained VGG-Face model was used to extract facial feature vectors from historical facial expression data.

[0013] Random noise ∈ is introduced into the standard normal distribution. A fully connected network is used to convert historical speech and facial feature vectors into mean and variance, respectively. Weighted average fusion is then used to fuse the mean and variance, yielding the fused mean and variance of the historical data. Based on the fused mean and variance, a variational distribution q(Z│F′)=U(μ,σ) is defined using a Gaussian distribution. 2), where U is the sign of the normal distribution, Z is the historical fusion feature vector, μ is the mean after fusion, σ is the variance after fusion, and F′ is the speech and facial feature vector;

[0014] The historical fusion feature vector Z is generated using a reparameterized formula;

[0015] The prior distribution p(Z) = U(0,I) is defined using the standard normal distribution, where I is the unit variance matrix;

[0016] The KL divergence KL(q(Z│F′)|‖p(Z)) between the variational distribution q(Z│F′) and the prior distribution p(Z) is calculated using the following formula:

[0017]

[0018] Where K is the dimension of the historical fusion feature vector Z, and Let be the mean and variance of the variational distribution q(Z│F′) in the k-th dimension, respectively.

[0019] The KL divergence KL(q(Z│F′)‖p(Z)) is defined as the redundancy information loss function L. red The formula is:

[0020] L red =KL(q(Z│F′)|‖p(Z)),

[0021] The model parameters of KL divergence are optimized using the backpropagation algorithm, iteratively minimizing the redundant information loss function L. red ;

[0022] The cumulative variance contribution rate method is used to set the number of principal components 'a'. The covariance matrices of speech and facial feature vectors are calculated separately. The covariance matrices are decomposed to obtain the eigenvalues ​​and eigenvectors of speech and face. The eigenvalues ​​are sorted in descending order, and the eigenvectors corresponding to the first 'a' largest eigenvalues ​​are retained to form the principal component matrix. The principal component matrix is ​​used to project the speech and facial feature vectors to obtain the latent representations of speech and facial feature vectors.

[0023] The energy function E(Z,Y) is constructed using the energy baseline model, and the formula is as follows:

[0024] E(Z,Y)=-<Z,f(Y)> +b Z ,

[0025] Where Z is the historical fusion feature vector.<Z,f(Y)> The inner product of the historical fusion feature vector and the linear mapping function f(Y), where f(Y) is the linear mapping function, b Z This is the energy reference bias term;

[0026] The linear mapping function f(Y) is given by the following formula:

[0027] f(Y) = W y ·Y+b Y ,

[0028] Among them W y Let Y be the parameter matrix of the mapping, where Y is the sentiment label and b is the number of digits. Y For the bias term of the mapping;

[0029] Randomly extract historical fusion feature vectors and corresponding sentiment labels from the training set, and denote them as positive sample pairs. Randomly select another sentiment label from the historical fusion feature vectors extracted in the same batch, and denote it as a negative sample pair.

[0030] Based on contrastive learning, the energy difference loss function L is defined. info The formula is:

[0031]

[0032] Where N is the total number of samples, Z i Y' is the historical fusion feature vector of the i-th sample. i Let Y' be the sentiment label of the i-th sample. j ' is the incorrect sentiment label paired with the historical fusion feature vector of the i-th sample, exp-E(Z) i ′,Y i ′)) represents the energy term for positive samples, ∑ j≠i exp(-E(Z i ′,Y i ′)) represents the negative sample energy term;

[0033] The parameters of the energy function are optimized using gradient descent, iteratively minimizing the energy difference loss function L. info ;

[0034] Using multi-objective optimization, the redundant information loss function L is applied. red Sum of energy difference loss function L info Combined, it is defined as the information bottleneck loss function L. IB The formula is:

[0035] L IB =L info +α·L red ,

[0036] Where α is the tradeoff coefficient;

[0037] The mean and variance of the fused sample are iteratively optimized using a second-order optimization method.

[0038] The real-time collected multimodal data is used with a trained information bottleneck loss function L. IBThis yields the real-time fused feature vector z;

[0039] The real-time fused feature vector z is sorted according to time to form a time series z(t).

[0040] As a preferred embodiment of the mental health early warning method for mental health education described in this invention, the complexity of calculating the trajectory by constructing phase space trajectory points using delayed embedding refers to the optimal delay time τ for calculating the time series using the mutual information method.

[0041] The embedding dimension m is calculated using the FNN method;

[0042] Based on the optimal delay time τ and the embedding dimension m, the phase space trajectory point X(t) is constructed using delayed embedding;

[0043] The phase space trajectory points X(t) are sorted in time order to form the trajectory sequence {X(t)}. i )}=

[0044] {X(t1),X(t2),…,X(t A )}, where A is the total number of trajectory points in phase space, t i This refers to the i-th time point in the time series.

[0045] Connect each point in the phase space trajectory sequence sequentially in time order to form a closed polygon, with vertices {X(t1), X(t2), ..., X(t...}}. A The area S of the trajectory is calculated using the polygon area formula.

[0046] The trajectory length L of adjacent phase space trajectory points is calculated using the Euclidean distance accumulation method.

[0047] The local curvature of adjacent phase space trajectory points is calculated using the local curvature formula.

[0048] Calculate the local curvature C i The average value yields the total curvature C. avg ;

[0049] Using trajectory length L, trajectory area S, and total curvature C avg The combination of these factors defines the trajectory complexity D, which is expressed by the formula:

[0050]

[0051] Where D represents the trajectory complexity.

[0052] As a preferred embodiment of the mental health early warning method for mental health education described in this invention, wherein: the weighted summation calculation of mental health score and early warning index is set to a cluster number of 3, and the K-means clustering algorithm is used to classify the emotional state of trajectory complexity D, including stable, fluctuating and drastic fluctuation;

[0053] The percentage of time spent in emotional state was calculated using statistical analysis of the classification results. i ;

[0054] Calculate the emotional state entropy value H using the information entropy formula;

[0055] The mental health score R is calculated using a weighted summation based on the trajectory complexity D and the emotional state entropy value H.

[0056] The normal distribution method is used to set a health threshold ε. The mental health score R is compared with the health threshold ε. If R>ε, it is judged as high risk, an early warning is issued and the monitor is notified via email. If R≤ε, it is judged as normal and multimodal data monitoring continues.

[0057] As a preferred embodiment of the mental health early warning method for mental health education described in this invention, the step of constructing a visual interface to display mental health scores in real time refers to using the front-end framework Vue.js to construct a visual interface to display mental health scores in real time, and displaying a risk warning icon at the top of the visual interface. When a warning is issued, the warning icon turns red and flashes.

[0058] Users who have passed real-name verification are allowed to view this information.

[0059] As a preferred embodiment of the mental health early warning method for mental health education described in this invention, the storage of the collected and analyzed multimodal data refers to storing the collected multimodal data and the analyzed mental health scores in a central database. The central database is sorted in chronological order and labeled with corresponding tags. The collected multimodal data and the analyzed mental health scores are backed up to the cloud simultaneously, and the integrity of the backup data is checked periodically.

[0060] Another object of the present invention is to provide a mental health early warning system for mental health education, comprising,

[0061] The collection and fusion module is used to collect multimodal data and preprocess it, and then extract the feature vectors of the preprocessed multimodal data for fusion.

[0062] The computational early warning module is used to calculate trajectory complexity by constructing phase space trajectory points using delayed embedding, and to calculate mental health scores and early warnings using weighted summation.

[0063] The visualization and storage module is used to build a visual interface to display mental health scores in real time and to store the multimodal data collected and analyzed.

[0064] A computer device includes: a memory and a processor; the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-mentioned mental health early warning method for mental health education.

[0065] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the aforementioned mental health early warning method for mental health education.

[0066] The beneficial effects of this invention are as follows: This invention collects and preprocesses multimodal data, extracts feature vectors from the preprocessed multimodal data and fuses them; it uses delayed embedding to construct phase space trajectory points to calculate trajectory complexity, and uses weighted summation to calculate mental health scores and early warnings; it enhances the sensitivity to fluctuations in emotional states, improves the accuracy and timeliness of mental health early warnings, and avoids the lag in monitoring mental states by users and monitors. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 A flowchart illustrating the mental health early warning methods for mental health education.

[0069] Figure 2 A schematic diagram of the structure of a mental health early warning system for mental health education. Detailed Implementation

[0070] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0071] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0072] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places throughout this specification does not necessarily refer to the same embodiment, nor is it a single embodiment or an embodiment selectively excluded from other embodiments.

[0073] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a mental health early warning method for mental health education. The mental health early warning method for mental health education includes,

[0074] S1. Collect multimodal data and preprocess it, then extract the feature vectors of the preprocessed multimodal data and fuse them.

[0075] Specifically, collecting and preprocessing multimodal data refers to using high-definition cameras and microphones to collect and preprocess users' multimodal data during mental health education.

[0076] The multimodal data includes voice and facial expression data;

[0077] The preprocessing includes adding timestamps to the collected multimodal data, performing time synchronization correction on the multimodal data using linear interpolation, removing noise from the speech data using a bandpass filter, standardizing the denoised speech data, removing noise from the facial expression data using Gaussian filtering, sharpening the facial expression data using the Laplacian operator, and aligning the facial expression data using affine transformation.

[0078] By adding timestamps to multimodal data and using linear interpolation to correct the time synchronization of the data, this invention ensures the temporal consistency of all data modalities, overcoming the problem of differences in sampling frequencies between different modalities. By removing noise from speech data through bandpass filters, this invention effectively reduces environmental noise and interference signals outside the speech frequency band, thereby improving the purity of speech data and enhancing the robustness of emotion recognition models to low-frequency and high-frequency interference, providing a more reliable data source for speech emotion analysis. The combined application of Gaussian filtering and the Laplacian operator can effectively remove noise and enhance edge information, making the details of facial expression features clearer. This not only helps to eliminate noise in facial images but also improves the resolution of micro-expressions, providing richer image information for subsequent emotion recognition and micro-expression analysis. By standardizing and aligning facial expression data through affine transformation, this invention solves the problem of feature offset caused by differences in angle or position during the acquisition process. The aligned facial features provide a basis for the dynamic analysis of facial expression changes, helping to reveal the fluctuations in an individual's mental health.

[0079] Furthermore, the feature vectors of the preprocessed multimodal data are extracted and fused, which means collecting historical multimodal data and performing preprocessing and feature vector extraction to generate a training set;

[0080] The pre-trained FER+ model was used to classify emotions in historical speech data and assign corresponding emotion labels, and the pre-trained OpenSMILE model was used to classify emotions in historical facial expression data and assign corresponding emotion labels.

[0081] The pre-trained OpenAI Whisper model was used to extract speech feature vectors from historical speech data, and the pre-trained VGG-Face model was used to extract facial feature vectors from historical facial expression data.

[0082] Random noise ∈ ~U(0,1) is introduced into the standard normal distribution. A fully connected network is used to convert historical speech and facial feature vectors into mean and variance, respectively. Weighted average fusion is then used to fuse the mean and variance, yielding the fused mean and variance of the historical data. Based on the fused mean and variance, a variational distribution q(Z│F′)=U(μ,σ) is defined using a Gaussian distribution. 2 ), where U is the sign of the normal distribution, Z is the historical fusion feature vector, μ is the mean after fusion, σ is the variance after fusion, and F′ is the speech and facial feature vector;

[0083] The historical fusion feature vector Z is generated using a reparameterization formula, as follows:

[0084] Z = μ + σ·∈

[0085] Where ∈ represents random noise;

[0086] The prior distribution p(Z) = U(0,I) is defined using the standard normal distribution, where I is the unit variance matrix, ensuring that the variance of each dimension is 1, thus avoiding the accumulation of unnecessary variance information in the features;

[0087] The KL divergence KL(q(Z│F′)‖p(Z)) between the variational distribution q(Z│F′) and the prior distribution p(Z) is calculated to measure the redundancy of irrelevant information in the fused features. The formula is:

[0088]

[0089] Where K is the dimension of the historical fusion feature vector Z, and Let be the mean and variance of the variational distribution q(Z│F′) in the k-th dimension, respectively.

[0090] KL divergence is mathematically rigorous and computable. The distribution obtained through KL divergence optimization is closer to the standard normal distribution. Although other metrics (such as JS divergence) can also be used to compare distributions, their convergence characteristics and sensitivity to model training are low, so they cannot achieve the same effect. By calculating KL divergence, the distribution of historical comprehensive features is made close to the standard normal distribution, ensuring the regularization of features. This method improves the stability of the model in multimodal emotion feature processing and avoids the problem of poor model generalization caused by feature bias or excessive dimensionality. The application of KL divergence not only improves the accuracy of comprehensive features, but also ensures the model's sensitivity and adaptability to emotional states, making the model more stable when dealing with complex emotional data.

[0091] The KL divergence KL(q(Z│F′)‖p(Z)) is defined as the redundancy information loss function L. red The formula is:

[0092] L red =KL(q(Z│F′)‖p(Z)),

[0093] The model parameters of KL divergence are optimized using the backpropagation algorithm, iteratively minimizing the redundant information loss function L. red ;

[0094] In the process of generating historical comprehensive features, reducing redundant information helps the model focus on key features. The redundancy loss function provides a direct means to optimize redundant information, ensuring the representativeness and simplicity of features. By defining the redundancy loss function, feature redundancy is quantified and minimized. In contrast, existing technologies usually do not consider feature redundancy. This improvement enables the generated historical comprehensive features to reduce redundancy while maintaining information content, providing a more concise and effective feature representation for subsequent emotion recognition. Compared with existing technologies, the method of this invention focuses more on improving the effectiveness of features and the robustness of the model through redundancy loss.

[0095] The cumulative variance contribution rate method is used to set the number of principal components 'a'. The covariance matrices of speech and facial feature vectors are calculated separately. The covariance matrices are decomposed to obtain the eigenvalues ​​and eigenvectors of speech and face. The eigenvalues ​​are sorted in descending order, and the eigenvectors corresponding to the first 'a' largest eigenvalues ​​are retained to form the principal component matrix. The principal component matrix is ​​used to project the speech and facial feature vectors to obtain the latent representations of speech and facial feature vectors.

[0096] The energy function is constructed using the energy baseline model, and the formula is as follows:

[0097] E(Z,Y)=-<Z,f(Y)> +b Z ,

[0098] Where Z is the historical fusion feature vector, and f(Y) is the linear mapping function.<Z,f(Y)> b is the inner product of the historical fusion feature vector Z and the linear mapping function f(Y), used to measure the similarity between positive and negative samples. Z This is the energy reference bias term, used to adjust the reference level of energy;

[0099] The energy function is the core mechanism for distinguishing positive and negative sample pairs. Through the energy benchmark model, the matching degree of different emotion labels and feature vectors can be effectively calculated, optimizing the accuracy and reliability of emotion classification. Using the inner product as the energy benchmark, the similarity information of emotion labels and feature vectors can be preserved in the low-dimensional feature space, ensuring high accuracy of emotion recognition. Compared with feature matching methods that directly rely on distance metrics, the energy benchmark model constructs the energy term through the inner product, reducing computational complexity and improving the model's processing speed. By optimizing the energy term, the model has a higher anti-interference ability when processing emotion label and feature vector matching, enhancing the robustness of the model in multimodal emotion recognition.

[0100] The linear mapping function f(Y) transforms the sentiment label into the feature space, and the formula is as follows:

[0101] f(Y) = W y ·Y+b Y ,

[0102] Among them W y Let Y be the parameter matrix of the mapping, where Y is the sentiment label and b is the number of digits. Y Adjust the baseline of the label mapping as the bias term of the mapping;

[0103] The introduction of a linear mapping function transforms emotion labels into feature vectors. This mapping enables effective energy measurement by fusing emotion labels with historical feature vectors. It features low computational complexity and seamless integration with energy baseline models, ensuring the model's efficiency and robustness. Unlike traditional methods for processing discrete emotion labels, this method converts discrete labels into continuous feature representations through linear mapping, allowing emotion labels to be better integrated into the feature space. This not only enhances the continuity of label expression but also improves discriminability in the feature space, optimizing the accuracy of emotion classification.

[0104] Randomly extract historical fusion feature vectors and corresponding sentiment labels from the training set, and denote them as positive sample pairs. Randomly select another sentiment label from the historical fusion feature vectors extracted in the same batch, and denote it as a negative sample pair.

[0105] Based on contrastive learning, the energy difference loss function L is defined. info This ensures that the energy of positive sample pairs is as low as possible and the energy of negative sample pairs is as high as possible, thereby achieving an effective distinction between positive and negative samples. The formula is:

[0106]

[0107] Where N is the total number of samples, Z i Y' is the historical fusion feature vector of the i-th sample. i Let Y' be the sentiment label of the i-th sample. j ' is the incorrect sentiment label paired with the historical fusion feature vector of the i-th sample, exp(-E(Z) i ′,Y i ′)) represents the energy term for positive samples. A negative exponential transformation is used to reduce the energy of positive samples, thereby enhancing the model's preference for positive samples. ∑ j≠ i exp(-E(Z i ′,Y i ′)) represents the energy term for negative samples. The energy terms of all negative sample pairs are summed to increase the energy of the negative samples.

[0108] The energy difference loss function is a core loss function based on contrastive learning, specifically designed to optimize the energy difference between positive and negative samples. By optimizing the energy term of positive samples and suppressing the energy term of negative samples, the model's preference for positive samples can be effectively enhanced, thereby improving the accuracy and robustness of emotion recognition. While other loss functions (such as the common cross-entropy loss) are suitable for classification problems, they cannot achieve the same effective energy optimization in contrastive learning. Therefore, in this case, only the energy difference loss function can achieve the optimal effect. Compared to traditional loss functions, this invention optimizes the energy difference between positive and negative samples through the energy difference loss function, strengthening the energy of positive samples and suppressing the energy of negative samples, thus enhancing the model's ability to distinguish positive samples and improving the accuracy of emotion recognition. In emotion recognition tasks, by optimizing the energy difference between positive and negative samples, the model can more accurately match emotion labels and fuse features, improving the model's adaptability to complex emotion data. This enhances the model's discriminative power among different emotion labels and feature vectors, increasing the model's sensitivity and adaptability to emotional states. This improvement makes the model more robust in emotion recognition tasks involving multimodal data, better able to cope with the diversity and complexity of emotional states.

[0109] The parameters of the energy function are optimized using gradient descent, iteratively minimizing the energy difference loss function L. info ;

[0110] Using multi-objective optimization, the redundant information loss function L is applied. red Sum of energy difference loss function L info Combined, it is defined as the information bottleneck loss function L. IB The formula is:

[0111] LIB =L info +α·L red ,

[0112] Where α is a trade-off coefficient that controls the degree of redundancy information compression;

[0113] By employing multi-objective optimization, the model achieves accurate extraction of effective features and effective compression of redundant information. This multi-objective loss definition ensures that the expressiveness and conciseness of features are balanced when extracting comprehensive features. Compared with existing technologies, the definition of the information bottleneck loss function combines the effectiveness and conciseness of features, ensuring that the extracted comprehensive features reduce redundancy while maintaining information integrity. Optimizing the information bottleneck loss function can yield more expressive features, thereby improving the accuracy of emotion recognition. By introducing a tradeoff coefficient, controllable compression of redundant information is achieved, enabling the feature vector to maintain a concise and stable feature representation while meeting the needs of expressing emotional states. Through the multi-objective optimized information bottleneck loss function, the model can dynamically adjust the complexity and expressiveness of features during the feature extraction process, adapting to different emotional states and diverse emotional data. Compared with the fixed feature extraction strategy of traditional methods, this flexibility makes the model more robust and reliable in complex scenarios.

[0114] The mean and variance of the fused sample are iteratively optimized using a second-order optimization method.

[0115] The real-time collected multimodal data is used with a trained information bottleneck loss function L. IB This yields the real-time fused feature vector z;

[0116] The real-time fused feature vector z is sorted according to time to form a time series z(t).

[0117] Preprocessing steps improve data accuracy and consistency through temporal correction and noise reduction, laying a solid foundation for emotion feature extraction and subsequent emotion prediction. This effectively enhances dataset quality and model adaptability. Weighted average fusion of speech and facial feature vectors helps integrate multimodal features, enabling the emotion recognition model to comprehensively evaluate features from both modalities, thus improving the accuracy of emotion classification. Weighted average fusion not only adapts to the contributions of speech and facial features under different emotional states but also achieves more flexible emotion expression measurement by controlling weights, ensuring the model's multimodal fusion effect. Introducing random noise from the standard normal distribution and generating a variational distribution through reparameterization ensures data diversity and model robustness. Without distorting the data, it improves the model's generalization performance in feature extraction, making the fused feature vectors more representative and enhancing sensitivity to emotional state fluctuations, thus supporting more accurate psychological state assessment. By using KL divergence as the redundancy loss function, the model can achieve the best results in the most... While minimizing redundant information, key features are retained, optimizing the representational power of multimodal fusion feature vectors, reducing feature redundancy, avoiding excessive reliance on invalid data, and improving the accuracy of emotion recognition, making it more suitable for assessing mental health. Retaining principal components effectively simplifies feature vectors, improving the model's computational efficiency and prediction accuracy, thus ensuring the model's real-time performance and applicability in practical applications. The introduction of the energy difference loss function makes the model more sensitive to distinguishing between positive and negative samples, reducing interference from erroneous emotion labels and enhancing the ability to identify true emotional states. The optimization process of contrastive learning effectively improves the model's robustness and classification accuracy, giving it higher recognition stability under different psychological states. The information bottleneck loss function achieves a balance between information compression and emotional feature representation through multi-objective optimization combined with redundant information and the energy difference loss function. The optimized fusion feature vector is more concise and accurate, avoiding interference from redundant information. The information bottleneck loss function enables the emotional feature vector to retain the core features of emotional fluctuations with limited information, providing a concise and reliable expression of emotional states for mental health assessment.

[0118] S2. Use delayed embedding to construct phase space trajectory points to calculate trajectory complexity, and use weighted summation to calculate mental health scores and early warnings;

[0119] Specifically, the computational complexity of using delayed embedding to construct phase space trajectory points refers to the optimal delay time τ for calculating the time series using the mutual information method, including...

[0120] Calculate the mutual information value I(τ) of the delay time, sort the mutual information values ​​I(τ) of the delay time in ascending order, and set the time corresponding to the minimum mutual information value I(τ) of the delay time as the optimal delay time τ.

[0121] The embedding dimension m is calculated using the FNN method;

[0122] Based on the optimal delay time τ and the embedding dimension m, the phase space trajectory point X(t) is constructed using delayed embedding, as shown in the formula:

[0123] X(t)=[z(t),z(t+τ),z(t+2τ),…,z(t+(m-1)τ)],

[0124] The phase space trajectory points X(t) are sorted in time order to form the trajectory sequence {X(t)}. i )}=

[0125] {X(t1),X(t2),…,X(t A )}, where A is the total number of trajectory points in phase space, t i This refers to the i-th time point in the time series.

[0126] Setting u as the y-axis, each point in the phase space trajectory sequence is connected sequentially in time to form a trajectory path that evolves over time. This path demonstrates the dynamic behavior of the system in phase space. The last time point X(t) in the time series... A Connecting X(t1) with the first time point X(t2), forms a closed polygon with vertices {X(t1), X(t2), ..., X(t3)}. A The area S of the trajectory is calculated using the polygon area formula. The formula is:

[0127]

[0128] Where X x (t i ) is the phase space trajectory point X(t) i The coordinates of U on the x-axis u (t i ) is the phase space trajectory point X(t) i The coordinates of the point on the u-axis;

[0129] The trajectory length L of adjacent phase space points is calculated using the Euclidean distance accumulation method, with the following formula:

[0130]

[0131] Where X j (t i ) is the phase space trajectory point X(t) iThe coordinate value in the j-th dimension;

[0132] The local curvature C of adjacent phase space trajectory points is calculated using the local curvature formula. i The formula is:

[0133]

[0134] Where × is the cross product of two vectors, representing the angle between them, used to measure the curvature of the trajectory, ‖(X(t) i+1 )-X(t i ))‖ and‖Xt i )-X(t i-1 )‖ are the phase space trajectory points X(t) i+1 ) and X(t i The Euclidean distance between ) and the phase space trajectory point X(t) i ) and X(t i-1 The Euclidean distance between them;

[0135] Calculate the local curvature C i The average value yields the total curvature C. avg The formula is:

[0136]

[0137] Using trajectory length L, trajectory area S, and total curvature C avg The combination of these factors defines the trajectory complexity D, which is expressed by the formula:

[0138]

[0139] Where D represents the trajectory complexity.

[0140] The trajectory complexity formula combines three major indicators: trajectory length, area, and curvature. By comprehensively considering fluctuation amplitude, spatial range, and frequency of change, it can comprehensively and accurately quantify the complexity of emotional states. Other simple features (such as trajectory length or area alone) cannot simultaneously take into account the intensity and drasticness of emotional state fluctuations, and therefore cannot achieve the same recognition effect. Curvature reflects the drasticness of trajectory changes and is of great significance in emotion fluctuation analysis. Relying solely on trajectory length or area will ignore the frequent fluctuations of emotions, while adding curvature can effectively make up for this deficiency, thereby comprehensively reflecting the dynamic changes of emotional states. This overcomes the limitations of traditional single-feature methods, making the quantification of emotion fluctuations more representative, improving the accuracy of emotion recognition models, helping to identify potential mental health risks earlier and more accurately, and making the model more stable and reliable when dealing with complex emotional data.

[0141] Optimal delay time avoids excessive redundant information while preserving key features in the time series, making the trajectory of emotional fluctuations clearer. This invention effectively captures the temporal characteristics of emotional fluctuations, thereby more accurately assessing changes in users' psychological states. It provides deep data support in the time dimension for mental health assessment. After determining a suitable embedding dimension, this invention can better display the dynamic evolution of emotional states in the reconstructed phase space, avoiding interference from false neighbors and making emotional state identification more accurate. The serialized trajectory structure intuitively reflects the amplitude and frequency of emotional state fluctuations, laying the foundation for subsequent quantification of trajectory complexity and providing a more reliable data source for real-time assessment of mental health. The longer the trajectory length, the more stable the emotional state. Greater fluctuations may reflect instability in psychological state. Trajectory length, as a measure of the intensity of emotional fluctuations, can help mental health monitoring systems accurately identify high-risk states and provide timely warnings. Larger trajectory areas usually indicate higher complexity of emotional states, which helps identify user states with intense or diverse emotional fluctuations. This invention quantifies the breadth of emotional fluctuations through area calculation, providing additional geometric features for mental health status assessment. Through curvature calculation, this invention can analyze the characteristics of emotional fluctuations from a geometric perspective, providing an important reference for mental health early warning. Trajectory complexity, as a core indicator of mental health assessment, provides an innovative method to assess the complexity of users' emotional states, possessing high practicality and scientific value.

[0142] Furthermore, a weighted summation method was used to calculate the mental health score and early warning index. Based on expert opinions, the number of clusters was set to 3. The K-means clustering algorithm was used to classify the emotional state by trajectory complexity D, including stable, fluctuating, and drastic fluctuation.

[0143] The percentage of time spent in emotional state was calculated using statistical analysis of the classification results. i The formula is:

[0144]

[0145] Where T i T represents the duration of the i-th emotional state. total Total observation time;

[0146] The emotional state entropy value H is calculated using the information entropy formula, which is:

[0147] H = -∑ i O i log(O i ),

[0148] The mental health score R is calculated using a weighted summation based on the trajectory complexity D and the emotional state entropy value H.

[0149] The normal distribution method is used to set a health threshold ε. The mental health score R is compared with the health threshold ε. If R>ε, it is judged as high risk, an early warning is issued and the monitor is notified via email. If R≤ε, it is judged as normal and multimodal data monitoring continues.

[0150] Classification methods can intuitively reflect different patterns of emotional fluctuations, providing a foundation for pattern recognition and analysis of emotional fluctuations. Statistical analysis calculates the time proportion of each emotional state, quantifying its duration and frequency. These indicators help identify the stability and trends of a user's emotional state. Calculating the entropy value of an emotional state using the information entropy formula quantifies its complexity and uncertainty. Entropy calculation helps accurately grasp the characteristics of emotional fluctuations, especially in cases of frequent fluctuations. The information entropy value can reveal the patterns of emotional fluctuations, thus aiding in a more comprehensive understanding of the user's psychological state. Emotional fluctuation amplitude, as a direct measure of intensity, provides crucial data support for mental health scoring. Quantifying fluctuation amplitude allows the system to objectively evaluate the intensity of emotional fluctuations, facilitating timely identification of abnormal emotional fluctuations. Mental health scoring is a quantitative representation of psychological state, providing users with an intuitive mental health assessment result. The introduction of mental health scoring simplifies the quantification process, enabling the system to quickly and accurately assess a user's psychological state. This early warning mechanism helps monitors intervene promptly to avoid potential mental health crises. The setting of health thresholds provides scientific judgment criteria, offering automated and efficient risk identification capabilities for mental health monitoring.

[0151] S3. Construct a visual interface to display mental health scores in real time, and store and analyze the collected multimodal data;

[0152] Specifically, building a visual interface to display mental health scores in real time refers to using the front-end framework Vue.js to build a visual interface that displays mental health scores in real time, and displaying a risk warning icon at the top of the visual interface. When a warning is issued, the warning icon turns red and flashes.

[0153] Users who have passed real-name verification are allowed to view this information.

[0154] Vue.js's responsive design ensures that data changes are instantly reflected on the interface, eliminating the need for users to manually refresh the page when viewing their mental health scores. This enhances the user experience. Compared to simple numerical displays, the graphical interface more vividly illustrates emotional fluctuation trends and changes in mental health scores, making it easy for non-professional users to understand the meaning of mental health scores and thus increasing their awareness of their own mental health. The flashing red indicator not only visually highlights risk states but also serves as a psychological cue, allowing users to intuitively perceive the urgency of their mental state, greatly improving the system's early warning effectiveness. It provides an instant feedback mechanism for mental health monitoring and intervention, ensuring that changes in mental health scores are displayed on the interface in real time. This avoids delays in monitoring mental health for both users and monitors, effectively preventing the leakage of mental health data and meeting privacy protection requirements, providing users with a higher level of trust. The interface design makes it easy for non-professional users to get started, reducing the difficulty of understanding mental health scores and making mental health monitoring functions more widespread and user-friendly. It also improves the response speed of mental health monitoring, providing more timely and scientific evidence for mental health intervention and preventing the exacerbation of mental health problems due to delayed early warnings.

[0155] Furthermore, storing the collected and analyzed multimodal data refers to storing the collected multimodal data and the generated mental health scores in a central database. The central database is sorted in chronological order and labeled accordingly. Simultaneously, the collected multimodal data and the generated mental health scores are backed up to the cloud, and the integrity of the backup data is checked regularly.

[0156] This approach not only improves data storage efficiency but also provides systematic data support for mental health research. It helps uncover potential patterns in emotional fluctuations during large-scale data analysis. Through this management method, mental health assessment systems can more accurately capture changes in users' emotional fluctuations, providing a data foundation for dynamic analysis and trend prediction of mental health scores. This helps identify potential mental health risks. Multiple backup storage ensures the system can quickly recover in case of emergencies, giving the mental health monitoring system a high level of data protection capabilities. This ensures the long-term stability of mental health score data, supports the system in obtaining highly reliable data during long-term tracking and analysis, and improves the overall data quality of mental health monitoring. The data security system protects mental health data from external attacks or hardware failures, ensuring the security and integrity of mental health score data. This provides reliable support for users' mental health management. The centralized management of large-scale data provides a solid foundation for mental health research and promotes the development of mental health assessment technology.

[0157] Example 2, refer to Figure 2This is the second embodiment of the present invention, which differs from the previous embodiment. It provides a mental health early warning system for mental health education, comprising:

[0158] The collection and fusion module is used to collect multimodal data and preprocess it, and then extract the feature vectors of the preprocessed multimodal data for fusion.

[0159] The computational early warning module is used to calculate trajectory complexity by constructing phase space trajectory points using delayed embedding, and to calculate mental health scores and early warnings using weighted summation.

[0160] The visualization and storage module is used to build a visual interface to display mental health scores in real time and to store the multimodal data collected and analyzed.

[0161] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0163] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0164] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

Claims

1. A method for early warning of mental health issues in mental health education, characterized by: include, Collect multimodal data and preprocess it, then extract the feature vectors of the preprocessed multimodal data and fuse them; The trajectory complexity is calculated by constructing phase space trajectory points using delayed embedding, and mental health scores and early warnings are calculated using weighted summation. The calculation of trajectory complexity using delayed embedding refers to calculating the optimal delay time for the time series using mutual information. ; The embedding dimension m is calculated using the FNN method; Based on optimal delay time Given the embedding dimension m, use delayed embedding to construct the phase space trajectory point X(t); The phase space trajectory points X(t) are sorted in time order to form a trajectory sequence. A represents the total number of trajectory points in phase space. This refers to the i-th time point in the time series. Connect each point in the phase space trajectory sequence sequentially in time to form a closed polygon, with the vertices of the polygon being... The area S of the trajectory is calculated using the polygon area formula; The trajectory length L of adjacent phase space trajectory points is calculated using the Euclidean distance accumulation method. The local curvature of adjacent phase space trajectory points is calculated using the local curvature formula. Calculate local curvature The average value yields the total curvature. ; Using trajectory length L, trajectory area S, and total curvature The combination of these factors defines the trajectory complexity D, which is expressed by the formula: ， Where D is the trajectory complexity; A visual interface is built to display mental health scores in real time, and to store the collected and analyzed multimodal data. A weighted summation method is used to calculate mental health scores and early warning indicators. The number of clusters is set to 3, and the K-means clustering algorithm is used to classify emotional states with trajectory complexity D, including stable, fluctuating, and drastic fluctuations. The percentage of time spent in emotional states was calculated using statistical analysis of the classification results. ; Calculate the emotional state entropy value H using the information entropy formula; The mental health score R is calculated using a weighted summation based on the trajectory complexity D and the emotional state entropy value H. Set health thresholds using the normal distribution method. The mental health score R was compared with the health threshold. To make a comparison, if If it is determined to be high-risk, an early warning will be issued and the monitor will be notified via email. If it is normal, continue monitoring of multimodal data.

2. The mental health early warning method for mental health education as described in claim 1, characterized in that: The collection and preprocessing of multimodal data refers to the use of high-definition cameras and microphones to collect and preprocess users' multimodal data during mental health education. The multimodal data includes voice and facial expression data; The preprocessing includes adding timestamps to the collected multimodal data, performing time synchronization correction on the multimodal data using linear interpolation, removing noise from the speech data using a bandpass filter, standardizing the denoised speech data, removing noise from the facial expression data using Gaussian filtering, sharpening the facial expression data using the Laplacian operator, and aligning the facial expression data using affine transformation.

3. The mental health early warning method for mental health education as described in claim 2, characterized in that: The step of extracting and fusing the feature vectors of the preprocessed multimodal data refers to collecting historical multimodal data, preprocessing it, and extracting feature vectors to generate a training set. The pre-trained FER+ model was used to classify emotions in historical speech data and assign corresponding emotion labels, and the pre-trained OpenSMILE model was used to classify emotions in historical facial expression data and assign corresponding emotion labels. The pre-trained OpenAI Whisper model was used to extract speech feature vectors from historical speech data, and the pre-trained VGG-Face model was used to extract facial feature vectors from historical facial expression data. Introducing random noise from the standard normal distribution A fully connected network is used to convert historical speech and facial feature vectors into mean and variance, respectively. A weighted average fusion method is then used to fuse the mean and variance to obtain the fused mean and variance of the historical data. Based on the fused mean and variance, a variational distribution is defined using a Gaussian distribution. Where U is the symbol for the normal distribution. For historical fusion feature vectors, The mean after fusion. The variance after fusion. For speech and facial feature vectors; The historical fusion feature vector Z is generated using a reparameterized formula; Define the prior distribution using the standard normal distribution. ,in It is a unit variance matrix; Calculate variational distribution With prior distribution KL divergence between The formula is: ， Where K is the dimension of the historical fusion feature vector Z, and Variational distributions The mean and variance in the k-th dimension; KL divergence Defined as the redundancy information loss function The formula is: ， The model parameters of KL divergence are optimized using the backpropagation algorithm, iteratively minimizing the redundant information loss function. ; The cumulative variance contribution rate method is used to set the number of principal components 'a'. The covariance matrices of speech and facial feature vectors are calculated separately. The covariance matrices are decomposed to obtain the eigenvalues ​​and eigenvectors of speech and face. The eigenvalues ​​are sorted in descending order, and the eigenvectors corresponding to the first 'a' largest eigenvalues ​​are retained to form the principal component matrix. The principal component matrix is ​​used to project the speech and facial feature vectors to obtain the latent representations of speech and facial feature vectors. Constructing an energy function using an energy benchmark model The formula is: ， Where Z is the historical fusion feature vector. For historical feature vectors and linear mapping functions The inner product, It is a linear mapping function. This is the energy reference bias term; Linear mapping function The formula is: ， in Let Y be the parameter matrix of the mapping, and Y be the emotion label. For the bias term of the mapping; Randomly extract historical fusion feature vectors and corresponding sentiment labels from the training set, and denote them as positive sample pairs. Randomly select another sentiment label from the historical fusion feature vectors extracted in the same batch, and denote it as a negative sample pair. Energy difference loss function defined based on contrastive learning. The formula is: ， Where N is the total number of samples. Let i be the historical fusion feature vector of the i-th sample. Let i be the emotion label of the i-th sample. For incorrect sentiment labels paired with the historical fusion feature vector of the i-th sample, The energy term for positive samples. This is the negative sample energy term; The parameters of the energy function are optimized using gradient descent, iteratively minimizing the energy difference loss function. ; Using multi-objective optimization to reduce redundant information loss function Sum of energy difference loss function Combined, it is defined as the information bottleneck loss function. The formula is: ， in For the weighting factor; The mean and variance of the fused sample are iteratively optimized using a second-order optimization method. The real-time collected multimodal data is used with a trained information bottleneck loss function. This yields the real-time fused feature vector z; The real-time fused feature vector z is sorted according to time to form a time series z(t).

4. The mental health early warning method for mental health education as described in claim 3, characterized in that: The construction of a visual interface to display mental health scores in real time refers to using the front-end framework Vue.js to build a visual interface that displays mental health scores in real time, and displays a risk warning icon at the top of the visual interface. When a warning is issued, the warning icon turns red and flashes. Users who have passed real-name verification are allowed to view this information.

5. The mental health early warning method for mental health education as described in claim 4, characterized in that: The storage of the collected and analyzed multimodal data refers to storing the collected multimodal data and the generated mental health scores in a central database. The central database is sorted in chronological order and labeled accordingly. The collected multimodal data and the generated mental health scores are backed up to the cloud simultaneously, and the integrity of the backup data is checked periodically.

6. A mental health early warning system for mental health education based on the mental health early warning method for mental health education according to any one of claims 1-5, characterized in that: include, The collection and fusion module is used to collect multimodal data and preprocess it, and then extract the feature vectors of the preprocessed multimodal data for fusion. The computational early warning module is used to calculate trajectory complexity by constructing phase space trajectory points using delayed embedding, and to calculate mental health scores and early warnings using weighted summation. The visualization and storage module is used to build a visual interface to display mental health scores in real time and to store the multimodal data collected and analyzed.

7. A computer device, comprising: Memory and processor; The memory stores a computer program, characterized in that: when the processor executes the computer program, it implements the steps of the mental health early warning method for mental health education as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the mental health early warning method for mental health education as described in any one of claims 1 to 5.

Images