Psychological analysis method and system based on multi-modal features

Abstract

The invention provides a psychological analysis method and system based on multi-modal features, and relates to the technical field of psychological assessment. The method comprises the following steps: synchronously obtaining voice data, video data, text data and physiological signal data of a user; respectively denoising the data to obtain each piece of denoised modal data; performing feature extraction on each modal data to obtain each modal feature; mapping each modal feature vector to a unified semantic space through an attention weighting mechanism to generate a fusion feature matrix; a trained psychological analysis model is adopted to perform multi-task learning on the fusion feature matrix, and emotion classification, pressure indexes and psychological health risk scores are synchronously output; and updating model parameters of the psychoanalysis model through an online incremental learning mechanism based on the real-time feedback data of the user and the newly added sample. Through accurate data denoising, efficient feature extraction and flexible model optimization, comprehensive and accurate analysis of the psychological state of the user is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of psychological assessment, and in particular to a psychological analysis method and system based on multimodal features. Background Art

[0002] Traditional psychological assessments primarily rely on clinical interviews and questionnaires, such as the SCL-90 Symptom Checklist and the Hamilton Depression Rating Scale. These methods suffer from subjectivity, long assessment cycles, and high context dependence, making real-time dynamic monitoring difficult. While some studies have attempted to use single-modality data (such as speech emotion analysis or facial expression recognition) for supplementary assessments, the representational capabilities of single-modality data are limited.

[0003] The existing technical solutions have the following key defects:

[0004] There is a lack of collaborative analysis mechanism for multimodal data, which fails to effectively integrate the complementary information of speech, vision, and physiological signals; traditional machine learning methods rely on artificial feature engineering and have difficulty capturing complex correlation patterns between physical and mental states; the model's generalization ability is insufficient and it is sensitive to interference factors such as individual differences, cultural background, and environmental noise; there is a lack of credibility assurance mechanism, and it is unable to provide explainable analysis results that meet clinical requirements. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a psychological analysis method and system based on multimodal features, which realizes a comprehensive and accurate analysis of the user's psychological state through precise data denoising, efficient feature extraction and flexible model optimization.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A psychological analysis method based on multimodal features, comprising:

[0008] Synchronously obtain the user's voice data, video data, text data and physiological signal data;

[0009] Denoising the voice data, the video data, the text data and the physiological signal data respectively to obtain denoised data of each modality;

[0010] Perform feature extraction on each modal data to obtain the features of each modal;

[0011] The feature vectors of each modality are mapped to a unified semantic space through the attention weighting mechanism to generate a fusion feature matrix;

[0012] Using a trained psychological analysis model to perform multi-task learning on the fused feature matrix, and simultaneously outputting emotion classification, stress index, and mental health risk score;

[0013] Based on the user's real-time feedback data and newly added samples, the model parameters of the psychological analysis model are updated through an online incremental learning mechanism.

[0014] Preferably, denoising is performed on the voice data, the video data, the text data and the physiological signal data respectively to obtain denoised modal data, including:

[0015] Separating the speech and background noise in the speech data using a multi-channel spectral subtraction technique to obtain denoised speech data;

[0016] Denoising the video data using a non-local means algorithm to retain detailed information of facial expressions, thereby obtaining denoised video data;

[0017] Based on the word embedding model, a clustering algorithm is used to identify and remove irrelevant noise data in the text data to obtain denoised text data;

[0018] The physiological signal data is denoised using a wavelet transform combined with a reconstruction algorithm. By decomposing the signal, low-frequency components are extracted and high-frequency noise is removed to obtain denoised physiological signal data. The physiological signal data includes: heart rate variability signals, skin electrical response signals and electroencephalogram signals.

[0019] Preferably, the physiological signal data is denoised by applying a wavelet transform combined with a reconstruction algorithm, and the denoised physiological signal data is obtained by decomposing the signal to extract low-frequency components and remove high-frequency noise, including:

[0020] Performing time-domain normalization processing on the heart rate variability signal by aligning the R wave peak values to obtain a preprocessed heart rate variability signal;

[0021] Performing Butterworth low-pass filtering on the galvanic skin response signal to eliminate myoelectric interference to obtain a preprocessed galvanic skin response signal;

[0022] Performing power frequency notching and ICA artifact removal processing on the electroencephalogram signal to obtain a preprocessed electroencephalogram signal;

[0023] The preprocessed heart rate variability signal is decomposed into 5 layers, retaining the detail coefficients of the 3rd to 5th layers. The preprocessed galvanic skin response signal is decomposed into 7 layers, extracting the approximate coefficients of the 4th to 7th layers. The preprocessed EEG signal is then adaptively decomposed according to the frequency band characteristics. The δ wave is decomposed into 6 layers, and the γ wave is decomposed into 4 layers.

[0024] The decomposed subband coefficients are subjected to noise suppression, and the decomposition coefficients of the target frequency band are selected for inverse wavelet transform to obtain the reconstructed physiological signal data. Among them, the cD3-cD5 detail coefficients and cA5 approximate coefficients are retained when reconstructing the heart rate variability signal; only the cA4-cA7 approximate coefficients are used when reconstructing the skin electrical response signal; the EEG signal is separated and reconstructed by frequency band and then time-frequency domain fusion is performed.

[0025] Preferably, the calculation formula of the reconstructed physiological signal data is:

[0026]

[0027] in, is the reconstructed physiological signal data, J: the preset maximum number of decomposition layers, set according to the signal type, 5 for heart rate variability signal, 7 for galvanic skin response signal, and 4 to 6 for EEG signal; is the dynamic weight matrix, which is determined by the signal sparsity parameter Calculated, N is the signal length; α j is the detail coefficient retention factor, SNR j is the signal-to-noise ratio of the jth layer; β j is the nonlinear correction term coefficient, μ j is the mean value of detail coefficient, σ j is the standard deviation, τ is the adaptive threshold; is a nonlinear activation function defined as cD j (t) is the detail coefficient matrix after denoising at the jth layer, cA J (t) is the J-th layer approximation coefficient vector, K is the low-frequency gain factor,

[0028] Preferably, each modal feature includes: acoustic features, facial micro-expressions, body movement features, semantic emotion features and physiological response features.

[0029] Preferably, the extraction of facial micro-expression features is obtained by using a 3D-CNN network to analyze the spatiotemporal characteristics of facial muscle movements; the extraction of semantic emotional features is obtained by combining a BERT pre-training model and a domain adaptation layer to generate a context-aware emotion vector.

[0030] Preferably, the expression of the fusion feature matrix is:

[0031]

[0032] Where M is the total number of modalities, M = 4, corresponding to speech / visual / text / physiological modalities; is the characteristic matrix of the mth mode, Dm is the characteristic dimension of the mth mode, T is the preset time step, is the modal dynamic weight matrix, Q m is the quality assessment vector of the modal feature, K is the global context key vector, d is the scaling factor, d = 64, is the cross-modal interaction matrix, σ(·) is the improved nonlinear activation function, σ(x)=ReLU(x)+α·Sin(βx), α is the periodic term weight, α=0.1, β is the frequency adjustment factor, β=5, γ∈[0,1] is the modal credibility attenuation factor, ε m is the signal quality entropy of the mth mode, is a feature selection mask matrix, which is used to retain feature dimensions that are significantly related to the target psychological state based on a mutual information threshold τ, where τ=0.7.

[0033] Preferably, the psychological analysis model includes a shared feature extractor and a task-specific branch; the loss function of the psychological analysis model adopts a weighted multi-task loss; the weighted multi-task loss includes cross entropy loss, mean square error loss and contrast loss terms.

[0034] A psychological analysis system based on multimodal features, comprising:

[0035] A data acquisition unit, used to synchronously acquire the user's voice data, video data, text data and physiological signal data;

[0036] a data denoising unit, configured to denoise the voice data, the video data, the text data and the physiological signal data respectively to obtain denoised data of each modality;

[0037] A feature extraction unit is used to extract features from each modal data to obtain features of each modal;

[0038] The matrix fusion unit is used to map the feature vectors of each modality to a unified semantic space through an attention weighting mechanism to generate a fused feature matrix;

[0039] a psychological analysis unit, configured to perform multi-task learning on the fused feature matrix using a trained psychological analysis model, and simultaneously output emotion classification, stress index, and mental health risk score;

[0040] The model updating unit is used to update the model parameters of the psychological analysis model through an online incremental learning mechanism based on the user's real-time feedback data and newly added samples.

[0041] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0042] The present invention provides a multimodal feature-based psychological analysis method and system. The method comprises: synchronously acquiring a user's voice data, video data, text data, and physiological signal data; denoising each of the voice data, video data, text data, and physiological signal data to obtain denoised data for each modality; extracting features from each modality to obtain features for each modality; mapping the feature vectors of each modality to a unified semantic space using an attention weighting mechanism to generate a fused feature matrix; performing multi-task learning on the fused feature matrix using a trained psychological analysis model to simultaneously output an emotion classification, a stress index, and a mental health risk score; and updating the model parameters of the psychological analysis model using an online incremental learning mechanism based on real-time user feedback data and newly added samples. Through precise data denoising, efficient feature extraction, and flexible model optimization, the present invention achieves a comprehensive and accurate analysis of a user's psychological state. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the system structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] The purpose of the present invention is to provide a psychological analysis method and system based on multimodal features, which realizes a comprehensive and accurate analysis of the user's psychological state through precise data denoising, efficient feature extraction and flexible model optimization.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] Figure 1A flow chart of the method provided in the embodiment of the present invention is shown in FIG. Figure 1 As shown, the present invention provides a psychological analysis method based on multimodal features, comprising:

[0050] Step 100: Synchronously obtain the user's voice data, video data, text data and physiological signal data;

[0051] Step 200: De-noising the voice data, video data, text data and physiological signal data respectively to obtain de-noised data of each modality;

[0052] Step 300: extracting features from each modal data to obtain features of each modal;

[0053] Step 400: Mapping the feature vectors of each modality to a unified semantic space through an attention weighting mechanism to generate a fusion feature matrix;

[0054] Step 500: Use the trained psychological analysis model to perform multi-task learning on the fusion feature matrix, and simultaneously output emotion classification, stress index, and mental health risk score;

[0055] Step 600: Based on the user's real-time feedback data and newly added samples, the model parameters of the psychological analysis model are updated through an online incremental learning mechanism.

[0056] Specifically, in step 100, this embodiment integrates multiple sensing technologies and devices to achieve the synchronous acquisition of the user's voice data, video data, text data, and physiological signal data. First, the voice data is collected through a high-sensitivity microphone array, which can effectively capture the user's voice signal. At the same time, the graphical user interface (GUI) guides the user to perform standardized pronunciation exercises to ensure the consistency and reliability of the data. During the recording process, the microphone array processes the collected audio signal in real time and converts the data into a digital audio stream for subsequent acoustic feature extraction and denoising.

[0057] Video data is acquired using a high-resolution camera capable of recording the user's facial expressions and body movements at a rate of at least 60 frames per second. To achieve higher capture accuracy, the camera uses infrared lighting technology to ensure clear image capture even in low-light environments. Furthermore, the camera is directly connected to a computer or mobile device, performing real-time video processing via dedicated software and synchronously storing the encoded video stream in a database. During this process, the system monitors video quality to ensure that the captured data meets the expected standards for clarity and smoothness, facilitating subsequent facial expression analysis and feature extraction.

[0058] During the synchronization process, the operating system uses a standard timestamp mechanism to ensure that all modal data are strictly aligned in time. Specifically, the system sets a unified acquisition timestamp for each data type and marks the data with a timestamp to ensure that the data of each modality is in the same time base during subsequent analysis. In addition, the user's input text (for example, the emotional state or reaction entered in the application) will be captured through the embedded text input box, and the text data will be directly recorded and stored, and timestamp-marked together with the audio, video and physiological signal data to achieve comprehensive data synchronization. The collection of physiological signal data is completed through somatosensory sensors. The sensors can monitor the user's heart rate variability (HRV), galvanic skin response (GSR) and electroencephalogram (EEG) and other physiological indicators in real time. These indicators will be recorded synchronously during data collection to support subsequent psychological state analysis.

[0059] Through the above method, step 100 of this embodiment can effectively and accurately obtain various types of data simultaneously, laying a solid foundation for subsequent multimodal feature extraction and psychological analysis.

[0060] Preferably, denoising is performed on the voice data, the video data, the text data and the physiological signal data respectively to obtain denoised modal data, including:

[0061] Separating the speech and background noise in the speech data using a multi-channel spectral subtraction technique to obtain denoised speech data;

[0062] Denoising the video data using a non-local means algorithm to retain detailed information of facial expressions, thereby obtaining denoised video data;

[0063] Based on the word embedding model, a clustering algorithm is used to identify and remove irrelevant noise data in the text data to obtain denoised text data;

[0064] The physiological signal data is denoised using a wavelet transform combined with a reconstruction algorithm. By decomposing the signal, low-frequency components are extracted and high-frequency noise is removed to obtain denoised physiological signal data. The physiological signal data includes: heart rate variability signals, skin electrical response signals and electroencephalogram signals.

[0065] Specifically, in this embodiment, different denoising technologies are used for voice data, video data, text data, and physiological signal data to ensure the high quality and reliability of each modality of data. For the voice data, multi-channel spectral subtraction technology is used to separate the voice signal from the background noise. The system uses a microphone array to obtain audio data from multiple channels, and then calculates the spectral characteristics of the background noise part by performing spectral analysis on each channel. Then, using these estimated values of background noise, the corresponding noise spectrum is subtracted from the spectrum of the original voice signal to obtain the denoised voice data. This process is achieved through Fourier transform, and in order to ensure the naturalness of the voice signal, threshold limiting technology is applied to avoid the loss of useful information of the signal, so that the final voice quality reaches the best state.

[0066] The video data is denoised using a non-local means algorithm. This algorithm compares pixel values across adjacent frames and across regions within the same frame, assessing similarities and performing a weighted average to remove static background and video noise. During processing, the system evaluates the local structure of each pixel relative to its surrounding area, preserving detailed facial expressions and effectively removing noise introduced by low-light conditions. This process ensures that important features in the video data, such as facial micro-expressions and movements, are preserved after denoising, providing clear visual information for subsequent psychological analysis.

[0067] For text data, the implementation uses a clustering algorithm based on a word embedding model to identify and remove irrelevant noise data. Specifically, the system first preprocesses the text data, including word segmentation and stop word removal, and then maps the vocabulary into a high-dimensional space to form word embeddings. Through the clustering algorithm, the system can identify similarities between different text fragments and remove noise text that is irrelevant to emotional expression, such as typos and non-standard word usage, based on the actual application scenario. This process ensures that the denoised text data remains semantically consistent, achieving accurate sentiment analysis.

[0068] For physiological signal data, wavelet transform combined with a reconstruction algorithm is used for denoising. The specific implementation involves decomposing heart rate variability, galvanic skin response, and electroencephalogram (EEG) signals into subband coefficients of different frequency bands. During the denoising process, the system analyzes the frequency domain characteristics of the signals to extract low-frequency components and remove high-frequency noise. Heart rate variability signals are normalized in the time domain by aligning R-wave peaks, then decomposed into multiple layers of wavelet coefficients, and significant detail coefficients are selected for reconstruction. Galvanic skin response signals are similarly filtered using a Butterworth low-pass filter to eliminate myoelectric interference. EEG signals are artifact-removed using power frequency notching and independent component analysis (ICA). Finally, adaptive decomposition and reconstruction are performed based on the characteristics of each signal to obtain denoised physiological signal data, ensuring its suitability for subsequent psychological state analysis.

[0069] Preferably, the physiological signal data is denoised by applying a wavelet transform combined with a reconstruction algorithm, and the denoised physiological signal data is obtained by decomposing the signal to extract low-frequency components and remove high-frequency noise, including:

[0070] Performing time-domain normalization processing on the heart rate variability signal by aligning the R wave peak values to obtain a preprocessed heart rate variability signal;

[0071] Performing Butterworth low-pass filtering on the galvanic skin response signal to eliminate myoelectric interference to obtain a preprocessed galvanic skin response signal;

[0072] Performing power frequency notching and ICA artifact removal processing on the electroencephalogram signal to obtain a preprocessed electroencephalogram signal;

[0073] The preprocessed heart rate variability signal is decomposed into 5 layers, retaining the detail coefficients of the 3rd to 5th layers. The preprocessed galvanic skin response signal is decomposed into 7 layers, extracting the approximate coefficients of the 4th to 7th layers. The preprocessed EEG signal is then adaptively decomposed according to the frequency band characteristics. The δ wave is decomposed into 6 layers, and the γ wave is decomposed into 4 layers.

[0074] The decomposed subband coefficients are subjected to noise suppression, and the decomposition coefficients of the target frequency band are selected for inverse wavelet transform to obtain the reconstructed physiological signal data. Among them, the cD3-cD5 detail coefficients and cA5 approximate coefficients are retained when reconstructing the heart rate variability signal; only the cA4-cA7 approximate coefficients are used when reconstructing the skin electrical response signal; the EEG signal is separated and reconstructed by frequency band and then time-frequency domain fusion is performed.

[0075] Optionally, in this embodiment, the denoising process of the physiological signal data is performed by applying a wavelet transform combined with a reconstruction algorithm, and the specific steps are as follows: First, the heart rate variability signal is processed, and a time domain normalization method of R wave peak alignment is adopted to ensure the timing consistency of the signal. The system calibrates the starting and ending positions of the heart rate variability by detecting the R wave peak in the electrocardiogram, thereby eliminating the time offset caused by changes in acquisition conditions. The preprocessed heart rate variability signal is then subjected to a 5-layer wavelet decomposition, and the detail components of the signal are extracted using multi-scale analysis. During the decomposition process, the system retains the wavelet detail coefficients (cD3, cD4, cD5) of the 3rd to 5th layers and the approximate coefficient (cA5) of the 5th layer. These coefficients contain useful information of the heart rate variability signal and can be used for subsequent analysis.

[0076] For the galvanic skin response signal, this embodiment first uses a Butterworth low-pass filter to eliminate myoelectric interference. The Butterworth filter is designed to be 2nd order or higher, with a cutoff frequency determined experimentally, generally between 0.5Hz and 5Hz. The filtered signal is then subjected to a 7-layer wavelet decomposition. The system extracts the approximate coefficients of layers 4 to 7 (cA4, cA5, cA6, cA7). These coefficients are used to reconstruct the galvanic skin response data, preserving the long-term trend of the physiological signal and suppressing the impact of high-frequency noise on signal analysis.

[0077] The EEG signal is processed using power frequency notching and independent component analysis (ICA) methods for artifact removal. First, this embodiment sets a notch frequency of 60Hz and applies a band-stop filter to filter the signal to eliminate power supply noise. Then, the signal is analyzed based on its independent components, and the interference components in the EEG signal are separated through ICA to retain effective EEG activity. The preprocessed EEG signal is adaptively decomposed according to the frequency band characteristics. The δ wave uses a 6-layer decomposition, and the γ wave uses a 4-layer decomposition to ensure that the characteristics of each frequency band can be effectively extracted.

[0078] Next, this embodiment performs noise suppression on the decomposed subband coefficients to preserve important information in the signal. During the reconstruction phase, for heart rate variability signals, the system retains the detailed coefficients cD3, cD4, and cD5, along with the approximate coefficients cA5. For galvanic skin response signals, only the approximate coefficients cA4 through cA7 are used. The EEG signal is reconstructed based on the characteristics of each frequency band, followed by time-frequency fusion. This method integrates information from different frequency bands to produce a complete, denoised physiological signal dataset. Ultimately, the reconstructed physiological signal data is fed back into the psychological analysis model for further psychological state assessment and analysis.

[0079] Preferably, the calculation formula of the reconstructed physiological signal data is:

[0080]

[0081] in, is the reconstructed physiological signal data, J: the preset maximum number of decomposition layers, set according to the signal type, 5 for heart rate variability signal, 7 for galvanic skin response signal, and 4 to 6 for EEG signal; is the dynamic weight matrix, which is determined by the signal sparsity parameter Calculated, N is the signal length; α j is the detail coefficient retention factor, SNR j is the signal-to-noise ratio of the jth layer; β j is the nonlinear correction term coefficient, μ j is the mean value of detail coefficient, σj is the standard deviation, τ is the adaptive threshold; is a nonlinear activation function defined as cD j (t) is the detail coefficient matrix after denoising at the jth layer, cA J (t) is the J-th layer approximation coefficient vector, K is the low-frequency gain factor,

[0082] Optionally, in this embodiment, the calculation formula for the reconstructed physiological signal data adopts an innovative multi-level dynamic weighting method to achieve efficient denoising and signal reconstruction. This method combines different signal types (such as heart rate variability signals, skin electrical response signals, and electroencephalogram signals) with their characteristics to set the maximum number of decomposition layers, so that each signal can fully consider its unique frequency characteristics and noise characteristics during the processing. The introduction of the dynamic weight matrix, through the calculation of the signal sparsity parameter, can adjust the importance of the signal in real time during the reconstruction process, so that the parts with high noise and low signal quality can be properly suppressed, thereby reducing interference with the overall signal reconstruction.

[0083] Furthermore, the use of detail coefficient retention factors precisely controls the quality of signal reconstruction, ensuring that the retained detail components are optimally represented at the desired signal-to-noise ratio, thereby enhancing the model's sensitivity to weak signals. The design of nonlinear correction terms and adaptive thresholds further explores the intelligent development of signal processing, enhancing the algorithm's robustness in complex environments. In this way, the reconstructed physiological signal data not only faithfully reflects the user's true physiological state but also provides more reliable basic data for multimodal psychological analysis, providing stronger support for the application and research of clinical psychology.

[0084] Preferably, each modal feature includes: acoustic features, facial micro-expressions, body movement features, semantic emotion features and physiological response features.

[0085] Preferably, the extraction of facial micro-expression features is obtained by using a 3D-CNN network to analyze the spatiotemporal characteristics of facial muscle movements; the extraction of semantic emotional features is obtained by combining a BERT pre-training model and a domain adaptation layer to generate a context-aware emotion vector.

[0086] Furthermore, in this embodiment, the feature extraction process is carried out by conducting an in-depth analysis of information of different modalities. First, in terms of extracting acoustic features, this embodiment adopts a traditional acoustic feature extraction algorithm to extract features such as fundamental frequency, intonation, volume, and duration. Specifically, by performing a short-time Fourier transform (STFT) on the user's voice data, the system calculates the spectrogram of the voice, and then uses the Mel-frequency cepstral coefficient (MFCC) to extract more detailed acoustic features. These features can effectively reflect the user's emotional color and psychological state during the communication process, and provide basic data for subsequent sentiment analysis.

[0087] In order to extract facial micro-expression features, this embodiment uses a 3D convolutional neural network (3D-CNN) to perform spatiotemporal feature analysis. 3D-CNN can better capture the dynamic changes of facial muscles by considering the convolution operation of both time and space dimensions. In a specific implementation, the user's facial video will be segmented into a continuous sequence of image frames and input into the 3D-CNN model for training. Through multi-layer convolution and pooling operations, the model gradually extracts the characteristic information of facial muscle movement and generates a set of high-dimensional feature vectors. These facial micro-expression feature vectors can reflect the user's true emotional state and provide a more delicate and objective basis for psychological analysis.

[0088] In the process of extracting semantic sentiment features, this embodiment combines the BERT pre-trained model and the domain adaptation layer to obtain context-aware sentiment vectors. The system first normalizes the input text data, including steps such as word segmentation and stop word removal. The processed text is then input into the BERT model, which uses its powerful contextual understanding capabilities to generate floating embedding vectors for each word. These embedding vectors not only contain the semantic information of the word, but also capture the emotional tendency of the word in a specific context.

[0089] To further improve the accuracy of sentiment features, this embodiment integrates a domain adaptation layer, which fine-tunes the pre-trained model to better adapt it to the text characteristics of the target domain. The domain adaptation layer uses annotated data to enhance the sentiment vector output by BERT to improve its performance in specific contexts. Ultimately, after this series of processing, the semantic sentiment feature vector generated by the system will be imported into the subsequent psychological analysis module to help identify the user's emotional state, thereby fully supporting the formulation of mental health assessment and intervention strategies.

[0090] Preferably, the expression of the fusion feature matrix is:

[0091]

[0092] Where M is the total number of modalities, M = 4, corresponding to speech / visual / text / physiological modalities; is the characteristic matrix of the mth mode, D m is the characteristic dimension of the mth mode, T is the preset time step, is the modal dynamic weight matrix, Q m is the quality assessment vector of the modal feature, K is the global context key vector, d is the scaling factor, d = 64, is the cross-modal interaction matrix, σ(·) is the improved nonlinear activation function, σ(x)=ReLU(x)+α·Sin(βx), α is the periodic term weight, α=0.1, β is the frequency adjustment factor, β=5, γ∈[0,1] is the modal credibility attenuation factor, ε m is the signal quality entropy of the mth mode, is a feature selection mask matrix, which is used to retain feature dimensions that are significantly related to the target psychological state based on a mutual information threshold τ, where τ=0.7.

[0093] Specifically, in this embodiment, the expression of the fusion feature matrix embodies an innovative multimodal data integration strategy, which aims to deeply fuse various modalities (speech, vision, text, and physiological signals) through mathematical models. First, by setting the total number of modalities M and the feature matrix of each modality, the system can effectively organize and represent different data inputs and analyze the characteristics of each modality under a unified framework. The introduction of the modal dynamic weight matrix is particularly critical. It can evaluate the importance of each modality in the psychological state analysis in real time, so that the modality that performs well in a specific scenario can get a higher weight, thereby enhancing the accuracy of the overall analysis.

[0094] Furthermore, by utilizing a global context key vector and a cross-modal interaction matrix, the system can flexibly handle relationships between multiple modalities. By introducing an improved nonlinear activation function, the nonlinear interactions between features are enhanced, further improving the model's expressive power. The design of the periodic term and frequency adjustment factor enables the system to effectively address the time-varying characteristics of the signal and enhances the feature's sensitivity to dynamic changes in psychological state. By setting the modal credibility attenuation factor, the algorithm can intelligently reduce the influence of the corresponding modality when signal quality is poor, preventing noise from interfering with the analysis results. Finally, the application of a feature selection mask matrix ensures that the feature dimensions of the final output are closely related to the target psychological state through threshold selection based on mutual information, thereby greatly improving the effectiveness and reliability of the model. Overall, the construction of this fused feature matrix not only provides a solid foundation for complex psychological state analysis but also demonstrates cutting-edge technological innovation in multimodal data processing.

[0095] Preferably, the psychological analysis model includes a shared feature extractor and a task-specific branch; the loss function of the psychological analysis model adopts a weighted multi-task loss; the weighted multi-task loss includes cross entropy loss, mean square error loss and contrast loss terms.

[0096] Optionally, in this embodiment, the construction of the psychological analysis model adopts a structure that combines a shared feature extractor with task-specific branches. The shared feature extractor is designed to extract high-dimensional common features from the fused feature matrix, and the deep learning network can automatically identify feature representations that can be used for multi-task prediction. This part of the network effectively integrates data from different modalities through a combination of multi-layer convolution and fully connected layers, while ensuring that the extracted features can capture the potential correlation between the user's emotions, stress, and mental health status. The extracted features will be sent to multiple task-specific branches respectively, which independently process the emotion classification, stress index, and mental health risk score to ensure the accuracy of the output results and the optimization of the corresponding tasks.

[0097] During the training process, the model uses a weighted multi-task loss function to optimize the learning effect. Specifically, the loss function contains multiple loss terms, including cross-entropy loss (for emotion classification tasks), mean square error loss (for stress index prediction), and contrast loss term (for mental health risk score assessment). By weightedly combining each loss term, key tasks can gain greater influence in training, thereby improving overall performance. This process is optimized through the backpropagation algorithm, and advanced optimization methods such as the Adam optimizer are used to ensure the stability and convergence speed of the training process. After training is completed, the model can effectively identify the psychological states of different users and provide a scientific basis for subsequent applications.

[0098] After model training is complete, step 500 begins the multi-task learning process on the fused feature matrix. In this step, the feature vectors generated by the shared feature extractor are fed into different task-specific branches. These branches handle emotion classification, stress index prediction, and mental health risk assessment, respectively. Each branch is rigorously optimized based on its specific task objectives to ensure that the model output is as consistent as possible with the target effect.

[0099] When the model is running, it outputs the results of each task synchronously. The emotion classification model directly outputs the user's emotional state (such as happiness, anxiety, depression, etc.); the stress index uses a regression process to give a continuous stress score; and the mental health risk score outputs a risk level through a classifier. The model design allows each task to operate in parallel, making full use of shared feature information and improving processing efficiency. Therefore, the entire psychological analysis system not only has the ability to handle multiple psychological state monitoring tasks, but also can provide users with personalized mental health assessments and recommendations in real time through intelligent data interaction and multi-task learning, thereby effectively supporting people's mental health management.

[0100] Furthermore, in step 600 of this embodiment, the psychological analysis model is updated using an online incremental learning mechanism to promptly respond to users' real-time feedback data and newly added samples. The core concept of this mechanism is to gradually adjust and optimize model parameters by effectively integrating incoming new data without retraining the entire model. This improves computational efficiency and ensures the real-time performance of the model. During the online learning process, the system first receives users' real-time feedback data, including mood reports, stress perception, and mental health self-assessment results. This data is used as new training samples for the model to adaptively update.

[0101] Based on the existing model parameters, this embodiment uses new data as a small batch input and, through incremental learning, only slightly adjusts the relevant parameters in the model. In this process, it is very important to select an appropriate learning rate, which is dynamically adjusted according to the characteristics and importance of the new sample to ensure that the previously learned knowledge is not erased during the update. After preprocessing and feature extraction, the additional feedback data is input into a model such as a convolutional neural network, and optimization algorithms such as mini-batch stochastic gradient descent (SGD) are used to achieve incremental parameter updates, so that the model can maintain high performance under changing user needs and conditions.

[0102] During incremental learning, effective utilization of feedback data is key to improving model accuracy. Whenever new user feedback and samples are received, this embodiment first performs a data quality assessment to ensure that the input data meets certain validity criteria. This data is then processed through steps such as feature selection and recalibration to align it with the feature space of the existing model. Furthermore, to prevent catastrophic forgetting during model updates, the system employs an experience replay mechanism, regularly reviewing and retraining old samples. This ensures that the model can maintain its ability to absorb and utilize past knowledge while adapting to new patterns.

[0103] After updating the model parameters, this embodiment evaluates the performance of the new model, typically through cross-validation, testing the correspondence between user feedback and prediction results over different time periods. This step not only verifies the effectiveness of the update but also promptly identifies potential issues with the model, allowing for further adjustments when necessary. Through this online incremental learning mechanism, the psychological analysis model of this embodiment can adapt to changes in the user's psychological state at any time, enhancing its flexibility and effectiveness in practical applications and ensuring that users receive up-to-date, personalized support for mental health management.

[0104] Corresponding to the above method, such as Figure 2 As shown, this embodiment also provides a psychological analysis system based on multimodal features, including:

[0105] A data acquisition unit, used to synchronously acquire the user's voice data, video data, text data and physiological signal data;

[0106] a data denoising unit, configured to denoise the voice data, the video data, the text data and the physiological signal data respectively to obtain denoised data of each modality;

[0107] A feature extraction unit is used to extract features from each modal data to obtain features of each modal;

[0108] The matrix fusion unit is used to map the feature vectors of each modality to a unified semantic space through an attention weighting mechanism to generate a fused feature matrix;

[0109] a psychological analysis unit, configured to perform multi-task learning on the fused feature matrix using a trained psychological analysis model, and simultaneously output emotion classification, stress index, and mental health risk score;

[0110] The model updating unit is used to update the model parameters of the psychological analysis model through an online incremental learning mechanism based on the user's real-time feedback data and newly added samples.

[0111] The beneficial effects of the present invention are as follows:

[0112] (1) The present invention ensures that the various modal data obtained from speech, video, text and physiological signals are purer and more reliable by applying denoising techniques such as multi-channel spectral subtraction, non-local mean algorithm, clustering algorithm and wavelet transform, and significantly improves the accuracy of subsequent analysis.

[0113] (2) The present invention combines deep learning technologies (such as 3D-CNN and BERT models) to effectively capture key information in each modal data during the feature extraction stage, such as acoustic features, facial micro-expressions, and physiological response features, making the analysis of psychological state more accurate.

[0114] (3) Through the attention weighting mechanism, the present invention successfully maps the features of different modalities into a unified semantic space. The generated fusion feature matrix can fully reflect the user's psychological state and support the effectiveness of multi-task learning.

[0115] (4) The online incremental learning mechanism of the present invention can adapt to user feedback and new data in real time, promote the continuous improvement of the psychological analysis model, and ensure the effectiveness and accuracy of the analysis system in a changing environment.

[0116] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0117] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A psychological analysis method based on multimodal features, characterized in that: include: Synchronously obtain the user's voice data, video data, text data and physiological signal data; Denoising the voice data, the video data, the text data and the physiological signal data respectively to obtain denoised data of each modality; Perform feature extraction on each modal data to obtain the features of each modal; The feature vectors of each modality are mapped to a unified semantic space through the attention weighting mechanism to generate a fusion feature matrix; Using a trained psychological analysis model to perform multi-task learning on the fused feature matrix, and simultaneously outputting emotion classification, stress index, and mental health risk score; Based on the user's real-time feedback data and newly added samples, the model parameters of the psychological analysis model are updated through an online incremental learning mechanism.

2. The psychological analysis method based on multimodal features according to claim 1, characterized in that: Denoising the voice data, the video data, the text data, and the physiological signal data to obtain denoised modal data, including: Separating the speech from the background noise in the speech data using a multi-channel spectral subtraction technique to obtain denoised speech data; Denoising the video data using a non-local means algorithm to retain detailed information of facial expressions, thereby obtaining denoised video data; Based on the word embedding model, a clustering algorithm is used to identify and remove irrelevant noise data in the text data to obtain denoised text data; The physiological signal data is denoised using a wavelet transform combined with a reconstruction algorithm. By decomposing the signal, low-frequency components are extracted and high-frequency noise is removed to obtain denoised physiological signal data. The physiological signal data includes: heart rate variability signals, skin electrical response signals and electroencephalogram signals.

3. The psychological analysis method based on multimodal features according to claim 2, characterized in that: Applying wavelet transform combined with reconstruction algorithm to denoise the physiological signal data, extracting low-frequency components and removing high-frequency noise by decomposing the signal, and obtaining denoised physiological signal data, including: Performing time-domain normalization processing on the heart rate variability signal by aligning the R wave peak values to obtain a preprocessed heart rate variability signal; Performing Butterworth low-pass filtering on the galvanic skin response signal to eliminate myoelectric interference to obtain a preprocessed galvanic skin response signal; Performing power frequency notching and ICA artifact removal processing on the electroencephalogram signal to obtain a preprocessed electroencephalogram signal; The preprocessed heart rate variability signal is decomposed into 5 layers, retaining the detail coefficients of the 3rd to 5th layers. The preprocessed galvanic skin response signal is decomposed into 7 layers, extracting the approximate coefficients of the 4th to 7th layers. The preprocessed EEG signal is then adaptively decomposed according to the frequency band characteristics. The δ wave is decomposed into 6 layers, and the γ wave is decomposed into 4 layers. The decomposed subband coefficients are subjected to noise suppression, and the decomposition coefficients of the target frequency band are selected for inverse wavelet transform to obtain the reconstructed physiological signal data. Among them, the cD3-cD5 detail coefficients and cA5 approximate coefficients are retained when reconstructing the heart rate variability signal; only the cA4-cA7 approximate coefficients are used when reconstructing the skin electrical response signal; the EEG signal is separated and reconstructed by frequency band and then time-frequency domain fusion is performed.

4. The psychological analysis method based on multimodal features according to claim 3, characterized in that: The calculation formula of the reconstructed physiological signal data is: in, is the reconstructed physiological signal data, J: the preset maximum number of decomposition layers, set according to the signal type, 5 for heart rate variability signal, 7 for galvanic skin response signal, and 4 to 6 for EEG signal; is the dynamic weight matrix, which is determined by the signal sparsity parameter Calculated, N is the signal length; α j is the detail coefficient retention factor, SNR j is the signal-to-noise ratio of the jth layer; β j is the nonlinear correction term coefficient, μ j is the mean value of detail coefficient, σ j is the standard deviation, τ is the adaptive threshold; is a nonlinear activation function, defined as is the detail coefficient matrix after denoising at the jth layer, cA J (t) is the J-th layer approximation coefficient vector, K is the low-frequency gain factor, 5. The psychological analysis method based on multimodal features according to claim 1, characterized in that: The modal features include: acoustic features, facial micro-expressions, body movement features, semantic emotional features and physiological response features.

6. The psychological analysis method based on multimodal features according to claim 5, characterized in that: The facial micro-expression features are extracted by using a 3D-CNN network to analyze the spatiotemporal characteristics of facial muscle movements; the semantic emotion features are extracted by combining the BERT pre-training model and the domain adaptation layer to generate a context-aware emotion vector.

7. The psychological analysis method based on multimodal features according to claim 1, characterized in that: The expression of the fusion feature matrix is: Where M is the total number of modalities, M = 4, corresponding to speech / visual / text / physiological modalities; is the characteristic matrix of the mth mode, D m is the characteristic dimension of the mth mode, T is the preset time step, is the modal dynamic weight matrix, Q m is the quality assessment vector of the modal feature, K is the global context key vector, d is the scaling factor, d = 64, is the cross-modal interaction matrix, σ(·) is the improved nonlinear activation function, σ(x)=ReLU(x)+α·Sin(βx), α is the periodic term weight, α=0.1, β is the frequency adjustment factor, β=5, γ∈[0,1] is the modal credibility attenuation factor, ε m is the signal quality entropy of the mth mode, is a feature selection mask matrix, which is used to retain feature dimensions that are significantly related to the target psychological state based on a mutual information threshold τ, where τ=0.

7.

8. The psychological analysis method based on multimodal features according to claim 1, characterized in that: The psychological analysis model includes a shared feature extractor and a task-specific branch; the loss function of the psychological analysis model adopts a weighted multi-task loss; the weighted multi-task loss includes cross entropy loss, mean square error loss and contrast loss terms.

9. A psychological analysis system based on multimodal features, characterized in that: include: A data acquisition unit, used to synchronously acquire the user's voice data, video data, text data and physiological signal data; a data denoising unit, configured to denoise the voice data, the video data, the text data and the physiological signal data respectively to obtain denoised data of each modality; A feature extraction unit is used to extract features from each modal data to obtain features of each modal; The matrix fusion unit is used to map the feature vectors of each modality to a unified semantic space through an attention weighting mechanism to generate a fused feature matrix; a psychological analysis unit, configured to perform multi-task learning on the fused feature matrix using a trained psychological analysis model, and simultaneously output emotion classification, stress index, and mental health risk score; The model updating unit is used to update the model parameters of the psychological analysis model through an online incremental learning mechanism based on the user's real-time feedback data and newly added samples.

Images