Multi-modal emotion recognition method for depression tendency of teenagers

Abstract

The invention discloses a multi-mode emotion recognition method for the depression tendency of teenagers, and relates to the technical field of mental health monitoring and evaluation. Comprising the following steps: acquiring an EEG signal data set, an ECG signal data set, a voice signal data set and a text signal data set; the obtained data set is preprocessed; performing feature extraction and fusion on the preprocessed data set to obtain a multi-modal feature matrix; constructing a multi-modal emotion recognition model based on the combination of a deep neural network and an attention mechanism; and inputting the multi-modal feature matrix into a multi-modal emotion recognition model to obtain an emotion recognition result. Through visual output of a cross-modal attention mechanism, the transparency of the model is enhanced, and the user trust degree in a medical scene is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of mental health monitoring and assessment, and in particular to a multimodal emotion recognition method for adolescents with depressive tendencies. Background Art

[0002] Multimodal emotion recognition technology for adolescents with depressive tendencies has made significant progress in recent years, primarily by integrating multiple physiological and behavioral signals to improve recognition accuracy and robustness. Depression recognition methods based on bimodal fusion utilize electroencephalogram (EEG) and electrocardiogram (ECG) signals. High-precision portable sensors collect physiological data from adolescents in depressive-inducing scenarios. After preprocessing steps such as downsampling, wavelet transform, bandpass filtering, and normalization, high-level features from the EEG and ECG are extracted. EEG feature extraction utilizes a 2D-CNN network, while ECG feature extraction utilizes a CNN-LSTM network to capture temporal features and sequential dependencies, respectively. Subsequently, a weighted fusion is performed through the calculation of a modal cross-matrix and a softmax layer to form deep bimodal features. The fused features are then classified using an LSTM classifier to identify the level of depression. Multimodal emotion recognition systems further expand this approach, encompassing more physiological signals and data processing steps, ensuring effective feature fusion and classification across modalities. Review studies have shown that multimodal emotion recognition technology significantly improves emotion recognition accuracy by integrating multiple data sources, such as speech, text, facial expressions, and physiological signals, and leveraging deep learning techniques for feature extraction and classification. Data fusion strategies, such as early fusion, late fusion, and hybrid fusion, are widely used in multimodal emotion recognition. Each strategy has its own advantages and disadvantages and is suitable for different application scenarios. However, multimodal emotion recognition technology still faces numerous challenges in practical applications, including data redundancy and conflict, feature extraction complexity, missing data processing, real-time detection requirements, computational complexity, model interpretability, and cross-domain generalization. To address this, researchers have proposed various solutions, including developing efficient data fusion methods, leveraging deep learning to simplify feature extraction, developing robust data interpolation techniques, optimizing algorithms to meet real-time processing requirements, reducing computational complexity, improving model transparency and credibility, and leveraging transfer learning and domain adaptation techniques to enhance model generalization. Through these technologies and approaches, multimodal emotion recognition technology for adolescent depression is continuously developing and improving, providing strong support for early detection and intervention of adolescent depression.

[0003] Although existing technical solutions have made some progress in multimodal emotion recognition for adolescent depressive tendencies, there are still many technical defects and challenges. First, data redundancy and conflict issues are particularly prominent in multimodal emotion recognition, especially in the fusion process of audio and visual modalities. How to effectively deal with these redundant and conflicting data to improve recognition accuracy remains a difficult problem. Secondly, the complexity of feature extraction is also a key challenge. Especially when processing audio and physiological signals, extracting high-quality features requires complex algorithms and a large amount of computing resources, which places higher demands on real-time applications. In addition, the processing of missing data is also an issue that cannot be ignored. In practical applications, data missing due to sensor failure or user preferences is often encountered. How to maintain the robustness of the model in these cases is an important research direction.

[0004] The need for real-time detection also places higher demands on existing technologies. Existing systems often face computational complexity issues when processing multimodal data, making it difficult to meet the demands of real-time processing. Model interpretability is also a pressing issue. The black-box nature of deep learning models raises questions about their transparency and credibility in practical applications, especially in the medical and mental health fields, where model interpretability is crucial for user trust and acceptance. Insufficient cross-domain generalization capabilities limit the model's applicability to different datasets and application scenarios. Improving the model's generalization capabilities to adapt to different cultural backgrounds and language environments is also an important research topic.

[0005] In terms of emotion recognition technology, existing methods often struggle to balance the information fusion between modalities when processing data from different modalities, resulting in inaccurate classification results. In particular, in the identification of depressive tendencies in adolescents, the fusion of physiological signals (such as EEG and ECG) with behavioral signals (such as voice and facial expressions) requires more sophisticated processing to avoid information loss or redundancy. Furthermore, when processing multimodal data, emotion recognition technology often faces issues with data synchronization and time alignment, which is particularly critical in real-time applications. Finally, the lack of evaluation metrics and benchmark datasets for emotion recognition technology also limits its further development. A more comprehensive and standardized evaluation system is needed to facilitate comparison and verification between different studies.

[0006] Therefore, it is an urgent problem for those skilled in the art to propose a multimodal emotion recognition method for adolescent depressive tendencies to solve the difficulties existing in the existing technology. Summary of the Invention

[0007] In view of this, the present invention provides a multimodal emotion recognition method for adolescent depressive tendencies, which enhances model transparency and improves user trust in medical scenarios through the visual output of the cross-modal attention mechanism.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A multimodal emotion recognition method for adolescent depression, comprising the following steps:

[0010] S1. Acquire data: Acquire EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset;

[0011] S2. Data preprocessing: Preprocess the acquired EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset;

[0012] S3. Feature extraction and fusion: Extract and fuse features from the preprocessed EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset to obtain a multimodal feature matrix.

[0013] S4. Model Construction: Construct a multimodal emotion recognition model based on a deep neural network combined with an attention mechanism;

[0014] S5. Recognition results: Input the multimodal feature matrix into the multimodal emotion recognition model to obtain the emotion recognition results.

[0015] In the above method, optionally, the specific content of preprocessing the acquired EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset in S2 is:

[0016] S21. Preprocess the acquired EEG signal dataset:

[0017] S211. Perform bandpass filtering, signal segmentation, and common average reference value denoising on the EEG signal to obtain a processed EEG signal;

[0018] The formula for removing EEG noise using the common average reference value is:

[0019]

[0020] in, is the potential of the EEG signal after filtering at time t, x i (t) is the potential of the ith electrode at time t, G is the number of selected channels, and x j (t) is the electrode potential of a selected channel at time t;

[0021] S212. Perform Z-Score normalization on the processed EEG signal to obtain a normalized EEG signal:

[0022]

[0023] Among them, z is the score of the individual physiological signal obtained after standardization, is the mean of the segmented physiological signals of each person being tested, μ is the overall mean of the physiological signals of all persons being tested, N is the number of persons being tested, and x i The physiological signal of each person being tested;

[0024] S213. Converting the normalized EEG signal into a 2D image, and then inputting the 2D image into a 2D-CNN network for feature extraction;

[0025] S22. Preprocess the acquired ECG signal dataset:

[0026] S221. Downsample the ECG signal. The formula is:

[0027]

[0028] Where ω is the digital angular frequency of the ECG signal in the frequency domain, Y is the downsampled ECG signal, X is the original ECG signal, and L is the downsampling factor;

[0029] S222. Use wavelet transform to remove high-frequency noise, power frequency interference, and electromyography signals from the ECG signal. The formula is:

[0030]

[0031] Where λ is the selected noise reduction threshold, and P is the length of the ECG signal after downsampling;

[0032] S223. Segment the ECG signal to complete the ERP segmentation, remove the 1-second segments before and after each signal segment, and eliminate the influence of the edge filter;

[0033] S224. Perform Z-Score normalization on the ECG signal to obtain an ECG sequence output, which is used as input to the ECG feature extraction network;

[0034] S23. Preprocess the acquired speech signal dataset:

[0035] S231. Use spectral subtraction to remove background noise and interference;

[0036] S232. Extract the Mel-frequency cepstral coefficient of the speech signal to obtain the Mel-frequency cepstral coefficient feature. The formula of the Mel-frequency cepstral coefficient is:

[0037]

[0038] Where x(n) is the speech signal, w(n) is the window function, m is the Mel-frequency cepstral coefficient index, and N is the frame length;

[0039] S233. Normalize the Mel-frequency cepstral coefficient feature. The formula is:

[0040]

[0041] Among them, μ m and σ m are the mean and standard deviation of the Mel-frequency cepstral coefficient features;

[0042] S24. Preprocess the acquired text signal dataset:

[0043] S241. Segment the collected text into words or phrases according to semantics;

[0044] S242. Convert the segmented words or phrases into vector representations.

[0045] In the above method, optionally, the specific content of performing feature extraction on the preprocessed EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset in S3 is:

[0046] S31. Perform feature extraction on the preprocessed EEG signal dataset:

[0047] A ResNet-18 variant is used to extract high-level features of EEG signals. The network structure consists of four sets of two-dimensional convolutional networks. Each set of networks includes two convolutional layers, ReLU activation functions, and maximum pooling layers. Residual connections are added to prevent gradient vanishing.

[0048] F out =F in +F(F in ,W i )

[0049] Among them, F is the convolution operation, W i is a learnable parameter;

[0050] S32. Perform feature extraction on the preprocessed ECG signal dataset:

[0051] TCN is used to extract the temporal features and sequential dependencies of ECG signals. The network structure includes a one-dimensional convolution layer, BatchNormalization, ReLU layer, maximum pooling layer, and a two-layer LSTM network. Causal convolution is used to process temporal dependencies:

[0052] y t =ReLU(W k *x t-k,t+k +b k )

[0053] Among them, * is causal convolution, k is the expansion coefficient;

[0054] S33. Perform feature extraction on the preprocessed speech signal dataset:

[0055] The 1D-CNN network is used to extract the Mel-frequency cepstral coefficient features of the speech signal. The network structure includes a one-dimensional convolution layer, batch normalization, ReLU layer, and maximum pooling layer.

[0056] S34. Perform feature extraction on the preprocessed text signal dataset:

[0057] Use recurrent neural network to extract features from word vector sequences.

[0058] In the above method, optionally, in S3, the preprocessed EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset are subjected to feature fusion to obtain the deep multimodal features. Specifically, the features are as follows:

[0059] Calculate the modal cross matrix of the EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset after feature extraction

[0060] N EEG-ECG ,N EEG-Speech ,N EEG-Text ,N ECG-Speech ,N ECG-Text ,N Speech-Text , the formula is:

[0061]

[0062] Among them, F EEG ,F ECG ,F Speech ,F Text These are the high-level feature matrices for EEG, ECG, speech, and text respectively;

[0063] The correlation strength of each modality cross matrix is ​​calculated based on CoAttention through the cross-modal attention mechanism:

[0064]

[0065] Among them, q i 、k j are query and key vectors respectively, C ij represents the association strength between modes i and j;

[0066] The probability distribution values ​​α, β, γ, δ, ε, and ξ of each modal cross matrix are calculated through the Softmax layer. The formula is as follows:

[0067]

[0068] Finally, the weighted fusion multimodal feature matrix M is obtained, and the formula is as follows:

[0069] M=concatenate[αF EEG ,βF ECG ,γF Speech ,δF Speech ,εF Text ,ζF Text ]

[0070] Among them, concatenate is a concatenation function used to concatenate the four modal features to form deep multimodal features.

[0071] In the above method, optionally, the multimodal emotion recognition model based on the combination of deep neural network and attention mechanism constructed in S4 includes an input layer, a hidden layer, an attention layer and an output layer;

[0072] Among them, the input layer is used to receive the fused multimodal feature matrix M;

[0073] The hidden layer consists of multiple fully connected layers, each layer contains several neurons, and the activation function uses the Swish activation function. Among them, β is a learnable parameter;

[0074] Attention layer, used to calculate attention weight, the formula is:

[0075] e ij =v T tanh(W i h i +W j s j +b)

[0076] Among them, e ij is the attention score, v,W i ,W j ,b is a learnable parameter, h i is the input feature, s j is the context vector;

[0077] The attention score is then converted into attention weight through the Softmax function

[0078] The output layer uses BERT-based Decoder to output the depression level results. The formula is:

[0079] P(y|x)=softmax(W·BERT(h x )+b)

[0080] Among them, h x To fuse features, BERT captures global dependencies through self-attention.

[0081] In the above method, optionally, the emotion recognition results obtained in S5 include risk warning results, depression level results, and a visualization report.

[0082] In the above method, the loss function can be designed to adopt weighted cross-entropy loss, the formula is:

[0083]

[0084] Among them, w c is the category weight.

[0085] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a multimodal emotion recognition method for adolescent depressive tendencies, which has the following beneficial effects:

[0086] 1. Improved feature extraction capabilities: Using a ResNet-18 variant and TCN network to extract EEG and ECG features, respectively, this more efficiently captures the deep features of EEG and ECG signals and reduces information loss compared to traditional 2D-CNN and CNN-LSTM networks.

[0087] 2. Enhanced modal fusion: Introducing a cross-modal attention mechanism to dynamically adjust the weights of different modalities, resolve data redundancy and conflict issues, and achieve more accurate multimodal feature fusion (such as collaborative analysis of EEG, speech, and text);

[0088] 3. Improved classification accuracy: Using the BERT-based decoder as the classification head enhances the ability to process complex text information and multimodal interactions, significantly improving the accuracy of depression level recognition.

[0089] 4. Optimize network performance: Use the Swish activation function instead of ReLU to improve the model's nonlinear expression capabilities, accelerate convergence, and reduce computational complexity to meet real-time detection requirements;

[0090] 5. Solve the pain points of existing technologies:

[0091] (1) Data missing handling: Robust data interpolation techniques (such as Z-Score normalization) are used to address data missing issues caused by sensor failure or user preferences.

[0092] (2) Model interpretability: Through the visualization output of the cross-modal attention mechanism, the model transparency is enhanced and user trust in medical scenarios is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0094] Figure 1 A flowchart of a multimodal emotion recognition method for adolescent depression provided by the present invention;

[0095] Figure 2 This is a structural diagram of the multimodal emotion recognition model based on the combination of deep neural network and attention mechanism provided by the present invention. DETAILED DESCRIPTION

[0096] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0097] Reference Figure 1 As shown, the present invention discloses a multimodal emotion recognition method for adolescent depressive tendencies, comprising the following steps:

[0098] S1. Acquire data: Acquire EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset;

[0099] S2. Data preprocessing: Preprocess the acquired EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset;

[0100] S3. Feature extraction and fusion: Extract and fuse features from the preprocessed EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset to obtain a multimodal feature matrix.

[0101] S4. Model Construction: Construct a multimodal emotion recognition model based on a deep neural network combined with an attention mechanism;

[0102] S5. Recognition results: Input the multimodal feature matrix into the multimodal emotion recognition model to obtain the emotion recognition results.

[0103] Furthermore, the specific contents of preprocessing the acquired EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset in S2 are as follows:

[0104] S21. Preprocess the acquired EEG signal dataset:

[0105] S211. Perform bandpass filtering, signal segmentation, and common average reference value denoising on the EEG signal to obtain a processed EEG signal;

[0106] The formula for removing EEG noise using the common average reference value is:

[0107]

[0108] in, is the potential of the EEG signal after filtering at time t, x i (t) is the potential of the ith electrode at time t, G is the number of selected channels, and x j (t) is the electrode potential of a selected channel at time t;

[0109] S212. Perform Z-Score normalization on the processed EEG signal to obtain a normalized EEG signal:

[0110]

[0111] Among them, z is the score of the individual physiological signal obtained after standardization, is the mean of the segmented physiological signals of each person being tested, μ is the overall mean of the physiological signals of all persons being tested, N is the number of persons being tested, and x i The physiological signal of each person being tested;

[0112] S213. Converting the normalized EEG signal into a 2D image, and then inputting the 2D image into a 2D-CNN network for feature extraction;

[0113] S22. Preprocess the acquired ECG signal dataset:

[0114] S221. Downsample the ECG signal. The formula is:

[0115]

[0116] Where ω is the digital angular frequency of the ECG signal in the frequency domain, Y is the downsampled ECG signal, X is the original ECG signal, and L is the downsampling factor;

[0117] S222. Use wavelet transform to remove high-frequency noise, power frequency interference, and electromyography signals from the ECG signal. The formula is:

[0118]

[0119] Where λ is the selected noise reduction threshold, and P is the length of the ECG signal after downsampling;

[0120] S223. Segment the ECG signal to complete the ERP segmentation, remove the 1-second segments before and after each signal segment, and eliminate the influence of the edge filter;

[0121] S224. Perform Z-Score normalization on the ECG signal to obtain an ECG sequence output, which is used as input to the ECG feature extraction network;

[0122] S23. Preprocess the acquired speech signal dataset:

[0123] S231. Use spectral subtraction to remove background noise and interference;

[0124] S232. Extract the Mel-frequency cepstral coefficient of the speech signal to obtain the Mel-frequency cepstral coefficient feature. The formula of the Mel-frequency cepstral coefficient is:

[0125]

[0126] Where x(n) is the speech signal, w(n) is the window function, m is the Mel-frequency cepstral coefficient index, and N is the frame length;

[0127] S233. Normalize the Mel-frequency cepstral coefficient feature. The formula is:

[0128]

[0129] Among them, μ m and σ m are the mean and standard deviation of the Mel-frequency cepstral coefficient features;

[0130] S24. Preprocess the acquired text signal dataset:

[0131] S241. Segment the collected text into words or phrases according to semantics;

[0132] S242. Convert the segmented words or phrases into vector representations.

[0133] Furthermore, the specific content of feature extraction of the preprocessed EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset in S3 is as follows:

[0134] S31. Perform feature extraction on the preprocessed EEG signal dataset:

[0135] A ResNet-18 variant is used to extract high-level features of EEG signals. The network structure consists of four sets of two-dimensional convolutional networks. Each set of networks includes two convolutional layers, ReLU activation functions, and maximum pooling layers. Residual connections are added to prevent gradient vanishing.

[0136] F out =F in +F(F in ,W i )

[0137] Among them, F is the convolution operation, W i is a learnable parameter;

[0138] S32. Perform feature extraction on the preprocessed ECG signal dataset:

[0139] TCN is used to extract the temporal features and sequential dependencies of ECG signals. The network structure includes a one-dimensional convolution layer, BatchNormalization, ReLU layer, maximum pooling layer, and a two-layer LSTM network. Causal convolution is used to process temporal dependencies:

[0140] y t =ReLU(W k *x t-k,t+k +b k )

[0141] Among them, * is causal convolution, k is the expansion coefficient;

[0142] S33. Perform feature extraction on the preprocessed speech signal dataset:

[0143] The 1D-CNN network is used to extract the Mel-frequency cepstral coefficient features of the speech signal. The network structure includes a one-dimensional convolution layer, BatchNormalization, ReLU layer and maximum pooling layer;

[0144] S34. Perform feature extraction on the preprocessed text signal dataset:

[0145] Use recurrent neural network to extract features from word vector sequences.

[0146] Furthermore, in S3, the pre-processed EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset are subjected to feature fusion to obtain the following deep multimodal features:

[0147] Calculate the modal cross matrix of the EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset after feature extraction

[0148] N EEG-ECG ,N EEG-Speech ,NEEG-Text ,N ECG-Speech ,N ECG-Text ,N Speech-Text , the formula is:

[0149]

[0150] Among them, F EEG ,F ECG ,F Speech ,F Text They are high-level feature matrices of EEG, ECG, speech, and text respectively; the correlation strength of each modality cross matrix is ​​calculated based on CoAttention through the cross-modal attention mechanism

[0151]

[0152] Among them, q i 、k j are query and key vectors respectively, C ij represents the association strength between modes i and j;

[0153] The probability distribution values ​​α, β, γ, δ, ε, and ξ of each modal cross matrix are calculated through the Softmax layer. The formula is as follows:

[0154]

[0155] Finally, the weighted fusion multimodal feature matrix M is obtained, and the formula is as follows:

[0156] M=concatenate[αF EEG ,βF ECG ,γF Speech ,δF Speech ,εF Text ,ζF Text ]

[0157] Among them, concatenate is a concatenation function used to concatenate the four modal features to form deep multimodal features.

[0158] Furthermore, the multimodal emotion recognition model based on the combination of deep neural network and attention mechanism constructed in S4 includes input layer, hidden layer, attention layer and output layer;

[0159] Among them, the input layer is used to receive the fused multimodal feature matrix M;

[0160] The hidden layer consists of multiple fully connected layers, each layer contains several neurons, and the activation function uses the Swish activation function. Among them, β is a learnable parameter;

[0161] Attention layer, used to calculate attention weight, the formula is:

[0162] e ij =v T tanh(W i h i +W j s j +b)

[0163] Among them, e ij is the attention score, v,W i ,W j ,b is a learnable parameter, h i is the input feature, s j is the context vector;

[0164] The attention score is then converted into attention weight through the Softmax function

[0165] The output layer uses BERT-based Decoder to output the depression level results. The formula is:

[0166] P(y|x)=softmax(W·BERT(h x )+b)

[0167] Among them, h x To fuse features, BERT captures global dependencies through self-attention.

[0168] Furthermore, the emotion recognition results obtained in S5 include risk warning results, depression level results and visualization reports.

[0169] Furthermore, the loss function is designed to use weighted cross-entropy loss, the formula is:

[0170]

[0171] Among them, w c is the category weight.

[0172] In a specific embodiment, Figure 2 As shown, including the following:

[0173] (1) Selecting a dataset

[0174] 1.EEG signal datasets: DEAP Dataset, MAHNOB-HCI Dataset, etc.

[0175] 2. ECG signal datasets: PhysioNet Challenge 2017, MIT-BIH Arrhythmia Database, etc.

[0176] 3. Speech signal datasets: RAVDESSDataset, CREMA-D Dataset, etc.

[0177] 4. Text signal datasets: DAIC-WOZ Dataset, GoEmotions Dataset, etc.

[0178] (2) Data preprocessing

[0179] 1. Preprocess the acquired EEG signal dataset:

[0180] The EEG signal is subjected to band-pass filtering, signal segmentation and common average reference value denoising to obtain a processed EEG signal;

[0181] The formula for removing EEG noise using the common average reference value is:

[0182]

[0183] in, is the potential of the EEG signal after filtering at time t, x i (t) is the potential of the ith electrode at time t, G is the number of selected channels, and x j (t) is the electrode potential of a selected channel at time t;

[0184] Perform Z-Score normalization on the processed EEG signal to obtain the normalized EEG signal:

[0185]

[0186] Among them, z is the score of the individual physiological signal obtained after standardization, is the mean of the segmented physiological signals of each person being tested, μ is the overall mean of the physiological signals of all persons being tested, N is the number of persons being tested, and x i The physiological signal of each person being tested;

[0187] Convert the normalized EEG signal into a 2D image, and then input the 2D image into the 2D-CNN network for feature extraction;

[0188] 2. Preprocess the acquired ECG signal dataset:

[0189] The ECG signal is downsampled and the formula is:

[0190]

[0191] Where ω is the digital angular frequency of the ECG signal in the frequency domain, Y is the downsampled ECG signal, X is the original ECG signal, and L is the downsampling factor;

[0192] The high-frequency noise, power frequency interference and electromyography signal of ECG signal are removed by wavelet transform. The formula is:

[0193]

[0194] Where λ is the selected noise reduction threshold, and P is the length of the ECG signal after downsampling;

[0195] The ECG signal was segmented to complete the ERP segmentation, and the 1-second segments before and after each signal were removed to eliminate the influence of the edge filter;

[0196] Perform Z-Score normalization on the ECG signal (the same Z-Score normalization method as the EEG signal) to obtain the ECG sequence output, which is used as the input of the ECG feature extraction network;

[0197] 3. Preprocess the acquired speech signal dataset:

[0198] Spectral subtraction was used to remove background noise and interference;

[0199] Extract the Mel-frequency cepstral coefficient of the speech signal to obtain the Mel-frequency cepstral coefficient feature. The formula of the Mel-frequency cepstral coefficient is:

[0200]

[0201] Where x(n) is the speech signal, w(n) is the window function, m is the Mel-frequency cepstral coefficient index, and N is the frame length;

[0202] Normalize the Mel frequency cepstral coefficient features, the formula is:

[0203]

[0204] Among them, μ m and σ m are the mean and standard deviation of the Mel-frequency cepstral coefficient features;

[0205] 4. Preprocess the acquired text signal dataset:

[0206] Segment the collected text into words or phrases based on semantics, for example, using a dictionary-based word segmentation method or a deep learning word segmentation model;

[0207] Convert the segmented words or phrases into vector representations. Use word vector models such as Word2Vec and GloVe to map each word into a low-dimensional vector space for subsequent processing.

[0208] (3) Feature extraction and fusion

[0209] Feature extraction:

[0210] 1. Feature extraction of the preprocessed EEG signal dataset:

[0211] A ResNet-18 variant is used to extract high-level features of EEG signals. The network structure consists of four sets of two-dimensional convolutional networks. Each set of networks includes two convolutional layers, ReLU activation functions, and maximum pooling layers. Residual connections are added to prevent gradient vanishing.

[0212] F out =F in +F(F in ,W i )

[0213] Among them, F is the convolution operation, W i is a learnable parameter;

[0214] 2. Perform feature extraction on the preprocessed ECG signal dataset:

[0215] TCN is used to extract the temporal features and sequential dependencies of ECG signals. The network structure includes a one-dimensional convolution layer, BatchNormalization, ReLU layer, maximum pooling layer, and a two-layer LSTM network. Causal convolution is used to process temporal dependencies:

[0216] y t =ReLU(W k *x t-k,t+k +b k )

[0217] Among them, * is causal convolution, k is the expansion coefficient;

[0218] 3. Perform feature extraction on the preprocessed speech signal dataset:

[0219] The 1D-CNN network is used to extract the Mel-frequency cepstral coefficient features of the speech signal. The network structure includes a one-dimensional convolution layer, batch normalization, ReLU layer, and maximum pooling layer.

[0220] 4. Perform feature extraction on the preprocessed text signal dataset:

[0221] Use recurrent neural network to extract features from word vector sequence. Assume that the input word vector sequence is x1, x2, ..., x T , the hidden state is h t , then the calculation formula of LSTM is as follows:

[0222] Forget gate: f t =σ(W f [x t-1 ,h t-1 ]+b f ), where σ is the Sigmoid activation function, W f is the weight matrix of the forget gate, b f is the bias vector of the forget gate;

[0223] Input gate: i t =σ(W i [x t-1 ,h t-1 ]+b i ), where W i is the weight matrix of the input gate, b i is the bias vector of the input gate;

[0224] Candidate memory cells: C t =tanh(W C [x t-1 ,h t-1 ]+b C ), where W C is the weight matrix of candidate memory cells, b C is the bias vector of the candidate memory cell;

[0225] Memory cell renewal: Among them, f t is the output of the forget gate, C t-1 is the hidden state (memory cell) of the previous moment, i t is the output of the input gate, ⊙ is the Hadamard product (element-wise multiplication), are candidate memory cells;

[0226] Output gate: σ t =σ(W o [x t-1 ,h t-1 ]+b o ), where W o is the weight matrix of the output gate, b o is the bias vector of the output gate;

[0227] Hidden state update: h t =σ t ⊙tanh(C t ), where σ t is the output of the output gate;

[0228] Finally, the text feature vector representation is obtained.

[0229] Feature fusion:

[0230] Calculate the modal cross matrix of the EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset after feature extraction

[0231] N EEG-ECG ,N EEG-Speech ,N EEG-Text ,N ECG-Speech ,N ECG-Text ,N Speech-Text , the formula is:

[0232]

[0233] Among them, F EEG ,F ECG ,F Speech ,F Text These are the high-level feature matrices for EEG, ECG, speech, and text respectively;

[0234] The correlation strength of each modality cross matrix is ​​calculated based on CoAttention through the cross-modal attention mechanism:

[0235]

[0236] Among them, q i 、k j are query and key vectors respectively, C ij represents the association strength between modes i and j;

[0237] The probability distribution values ​​α, β, γ, δ, ε, and ξ of each modal cross matrix are calculated through the Softmax layer. The formula is as follows:

[0238]

[0239] Finally, the weighted fusion multimodal feature matrix M is obtained, and the formula is as follows:

[0240] M=concatenate[αF EEG ,βF ECG ,γF Speech ,δF Speech ,εF Text ,ζF Text ]

[0241] Among them, concatenate is a concatenation function used to concatenate the four modal features to form deep multimodal features.

[0242] Here it is assumed that the fusion of speech and text features adopts a weighted sum method, which can be adjusted according to actual conditions.

[0243] (4) Model training and classification

[0244] Model selection

[0245] Classification is performed using a model that combines a deep neural network (DNN) with an attention mechanism. DNNs have powerful nonlinear fitting capabilities and can learn complex feature representations; the attention mechanism allows the model to focus more on important features, improving classification accuracy.

[0246] Model structure

[0247] Input layer, used to receive the fused multimodal feature matrix M;

[0248] The hidden layer consists of multiple fully connected layers, each layer contains several neurons, and the activation function uses the Swish activation function. Among them, β is a learnable parameter;

[0249] Attention layer, used to calculate attention weight, the formula is:

[0250] e ij =v T tanh(W i h i +W j s j +b)

[0251] Among them, e ij is the attention score, v,W i ,W j ,b is a learnable parameter, h i is the input feature, s j is the context vector;

[0252] The attention score is then converted into attention weight through the Softmax function

[0253] The output layer uses BERT-based Decoder to output the depression level results. The formula is:

[0254] P(y|x)=softmax(W·BERT(h x )+b)

[0255] Among them, h x To fuse features, BERT captures global dependencies through self-attention.

[0256] The loss function is designed using weighted cross-entropy loss, the formula is:

[0257]

[0258] Among them, w c is the category weight.

[0259] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0260] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal emotion recognition method for adolescent depressive tendencies, characterized by: The following steps are involved: S1. Acquire data: Acquire EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset; S2. Data preprocessing: Preprocess the acquired EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset; S3. Feature extraction and fusion: Extract and fuse features from the preprocessed EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset to obtain a multimodal feature matrix. S4. Model Construction: Construct a multimodal emotion recognition model based on a deep neural network combined with an attention mechanism; S5. Recognition results: Input the multimodal feature matrix into the multimodal emotion recognition model to obtain the emotion recognition results.

2. A multimodal emotion recognition method for adolescent depressive tendencies according to claim 1, characterized in that: The specific contents of preprocessing the acquired EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset in S2 are as follows: S21. Preprocess the acquired EEG signal dataset: S211. Perform bandpass filtering, signal segmentation, and common average reference value denoising on the EEG signal to obtain a processed EEG signal; The formula for removing EEG noise using the common average reference value is: in, is the potential of the EEG signal after filtering at time t, x i (t) is the potential of the ith electrode at time t, G is the number of selected channels, and x j (t) is the electrode potential of a selected channel at time t; S212. Perform Z-Score normalization on the processed EEG signal to obtain a normalized EEG signal: Among them, z is the score of the individual physiological signal obtained after standardization, is the mean of the segmented physiological signals of each person being tested, μ is the overall mean of the physiological signals of all persons being tested, N is the number of persons being tested, and x i The physiological signal of each person being tested; S213. Converting the normalized EEG signal into a 2D image, and then inputting the 2D image into a 2D-CNN network for feature extraction; S22. Preprocess the acquired ECG signal dataset: S221. Downsample the ECG signal. The formula is: Where ω is the digital angular frequency of the ECG signal in the frequency domain, Y is the downsampled ECG signal, X is the original ECG signal, and L is the downsampling factor; S222. Use wavelet transform to remove high-frequency noise, power frequency interference, and electromyography signals from the ECG signal. The formula is: Where λ is the selected noise reduction threshold, and P is the length of the ECG signal after downsampling; S223. Segment the ECG signal to complete the ERP segmentation, remove the 1-second segments before and after each signal segment, and eliminate the influence of the edge filter; S224. Perform Z-Score normalization on the ECG signal to obtain an ECG sequence output, which is used as input to the ECG feature extraction network; S23. Preprocess the acquired speech signal dataset: S231. Use spectral subtraction to remove background noise and interference; S232. Extract the Mel-frequency cepstral coefficient of the speech signal to obtain the Mel-frequency cepstral coefficient feature. The formula of the Mel-frequency cepstral coefficient is: Where x(n) is the speech signal, w(n) is the window function, m is the Mel-frequency cepstral coefficient index, and N is the frame length; S233. Normalize the Mel-frequency cepstral coefficient feature. The formula is: Among them, μ m and σ m are the mean and standard deviation of the Mel-frequency cepstral coefficient features; S24. Preprocess the acquired text signal dataset: S241. Segment the collected text into words or phrases according to semantics; S242. Convert the segmented words or phrases into vector representations.

3. The multimodal emotion recognition method for adolescent depression according to claim 1, characterized in that: The specific content of feature extraction of the preprocessed EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset in S3 is as follows: S31. Perform feature extraction on the preprocessed EEG signal dataset: A ResNet-18 variant is used to extract high-level features of EEG signals. The network structure consists of four sets of two-dimensional convolutional networks. Each set of networks includes two convolutional layers, ReLU activation functions, and maximum pooling layers. Residual connections are added to prevent gradient vanishing. F out =F in +F(F in ,W i ) Among them, F is the convolution operation, W i is a learnable parameter; S32. Perform feature extraction on the preprocessed ECG signal dataset: TCN is used to extract the temporal features and sequential dependencies of ECG signals. The network structure includes a one-dimensional convolution layer, BatchNormalization, ReLU layer, maximum pooling layer, and a two-layer LSTM network. Causal convolution is used to process temporal dependencies: y t =ReLU(W k *x t-k,t+k +b k ) Among them, * is causal convolution, k is the expansion coefficient; S33. Perform feature extraction on the preprocessed speech signal dataset: The 1D-CNN network is used to extract the Mel-frequency cepstral coefficient features of the speech signal. The network structure includes a one-dimensional convolution layer, BatchNormalization, ReLU layer and maximum pooling layer; S34. Perform feature extraction on the preprocessed text signal dataset: Use recurrent neural network to extract features from word vector sequences.

4. The multimodal emotion recognition method for adolescent depression according to claim 1, characterized in that: In S3, the pre-processed EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset are subjected to feature fusion to obtain the following deep multimodal features: Calculate the modal cross matrix of the EEG signal dataset, ECG signal dataset, speech signal dataset, and text signal dataset after feature extraction N EEG-ECG ,N EEG-Speech ,N EEG-Text ,N ECG-Speech ,N ECG-Text ,N Speech-Text , the formula is: Among them, F EEG ,F ECG ,F Speech ,F Text These are the high-level feature matrices for EEG, ECG, speech, and text respectively; The correlation strength of each modality cross matrix is ​​calculated based on CoAttention through the cross-modal attention mechanism: Among them, q i 、k j are query and key vectors respectively, C ij represents the association strength between modes i and j; The probability distribution values ​​α, β, γ, δ, ε, and ξ of each modal cross matrix are calculated through the Softmax layer. The formula is as follows: Finally, the weighted fusion multimodal feature matrix M is obtained, and the formula is as follows: M=concatenate[αF EEG ,βF ECG ,γF Speech ,δF Speech ,εF Text ,ζF Text ] Among them, concatenate is a concatenation function used to concatenate the four modal features to form deep multimodal features.

5. A multimodal emotion recognition method for adolescent depressive tendency according to claim 1 or 4, characterized in that: The multimodal emotion recognition model constructed in S4 based on the combination of deep neural network and attention mechanism includes input layer, hidden layer, attention layer and output layer; Among them, the input layer is used to receive the fused multimodal feature matrix M; The hidden layer consists of multiple fully connected layers, each layer contains several neurons, and the activation function uses the Swish activation function. Among them, β is a learnable parameter; Attention layer, used to calculate attention weight, the formula is: e ij =v T tanh(W i h i +W j s j +b) Among them, e ij is the attention score, v,W i ,W j ,b is a learnable parameter, h i is the input feature, s j is the context vector; The attention score is then converted into attention weight through the Softmax function The output layer uses BERT-based Decoder to output the depression level results. The formula is: P(y|x)=softmax(W·BERT(h x )+b) Among them, h x To fuse features, BERT captures global dependencies through self-attention.

6. The multimodal emotion recognition method for adolescent depression according to claim 1, characterized in that: The emotion recognition results obtained in S5 include risk warning results, depression level results and visualization reports.

7. A multimodal emotion recognition method for adolescent depressive tendencies according to any one of claims 1 to 6, characterized in that: The loss function is designed using weighted cross-entropy loss, the formula is: Among them, w c is the category weight.

Images