Multimodal depression detection method based on spatio-temporal frequency domain enhancement and mutual attention mechanism

Abstract

The application discloses a multi-modal depression detection method based on space-time frequency domain enhancement and mutual attention mechanism, and specifically comprises the following steps: step 1, preprocessing of multi-modal perception data of depression subjects; step 2, joint space-time frequency domain feature mining; step 3, frequency spectrum gating guided speech enhancement; step 4, landmark guided visual semantic reconstruction; step 5, three-way parallel mutual attention cross-modal fusion; step 6, spatial enhancement and classification discrimination; step 7, sample set decision statistics and global index evaluation. The method solves the problems of weak facial and speech subtle feature capturing ability, insufficient video and audio cross-modal correlation mining and original data susceptible to background noise interference.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and affective computing technology, specifically involving a multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanisms. Background Technology

[0002] Depression is a common mental health disorder, and early screening and diagnosis are crucial for its prevention and treatment. Traditional diagnostic methods mainly rely on clinical interviews with doctors and self-rating scales, which are subject to problems such as high subjectivity, long processing time, inconvenience, and difficulty in large-scale coverage.

[0003] With the development of deep learning, automated depression detection based on facial expression and voice features has become a research hotspot. However, existing methods face the following challenges when processing multimodal data: Insufficient cross-modal correlation mining: Simple data splicing is insufficient to capture the complex complementary relationship between video facial expressions and voice intonation.

[0004] Weak ability to capture subtle features: Features related to depression are often hidden in subtle facial movements (such as eye micro-movements) and specific frequency bands of speech, and traditional methods are prone to losing these time-frequency details.

[0005] Noise interference: Raw speech and video data contain a large amount of background information that is irrelevant to emotion, which affects the accuracy of detection. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanism, which solves the problems of weak ability to capture subtle facial and speech features, insufficient mining of cross-modal correlations in audio and video, and the susceptibility of raw data to background noise interference.

[0007] The technical solution adopted in this invention is a multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanism, specifically including the following steps: Step 1: Preprocessing of multimodal perception data for subjects with depression; Step 2: Joint feature mining in the spatiotemporal and frequency domains; Step 3: Spectrum-gated guided speech enhancement; Step 4: Visual semantic reconstruction guided by landmarks; Step 5: Three-way parallel mutual attention cross-modal fusion; Step 6: Spatial augmentation and classification; Step 7: Decision statistics of sample set and evaluation of global indicators.

[0008] The invention is further characterized in that: Step 1 is as follows: Step 1.1: Visual and Landmark Feature Extraction: First, the facial video of the depressed subject is cropped into a sequence of N consecutive frames; the global visual feature vector of the image sequence is extracted using a pre-trained image feature extraction backbone network to obtain the visual sequence features; at the same time, the key point coordinates of the image sequence are extracted to obtain the landmark sequence. Step 1.2: Speech signal representation construction: After acquiring a single-channel audio signal that is completely synchronized with the video, perform a short-time Fourier transform and map the linear spectrum from high frequency to low frequency onto an M-channel Mel filter group to extract a Mel spectrogram sequence with a size equal to the number of audio time steps multiplied by M dimensions. Step 1.3: Unified Representation Space Mapping: Map the visual sequence, landmark sequence, and Mel spectrogram sequence extracted in Step 1.1 to the same latent feature dimension to generate initial visual sequence features, initial landmark sequence features, and initial audio features in a unified feature space.

[0009] Step 2 is as follows: Step 2.1: Temporal branch feature extraction: Input the visual sequence features and landmark sequence features extracted in Step 1.1 into the bidirectional gated recurrent unit, establish long-range temporal dependencies by forward and backward recursive calculations along the time axis, concatenate the forward hidden state vector and the backward hidden state vector, and calculate the mean along the time step to obtain the temporal feature representation of the macroscopic action trajectory. Step 2.2: Frequency domain branch feature extraction: The initial visual sequence features and initial landmark sequence features obtained in Step 1.3 are converted into complex spectrum matrices in the frequency domain using real-valued fast Fourier transform. The modulus of the complex spectrum matrix is ​​taken to obtain the spectrum amplitude map, and then the distribution is normalized and adjusted to the same feature dimension as the time domain branch output features in Step 2.1 through a linear projection layer to obtain the frequency domain feature representation. Step 2.3: Dynamic Residual Fusion: The temporal feature representation obtained in Step 2.1 is added and fused with the frequency domain feature representation multiplied by a preset weight coefficient. Then, the feature distribution is smoothed by layer normalization to output the spatiotemporal frequency domain enhanced feature representation, which includes enhanced visual features and enhanced landmark features. Through this step, the initial visual sequence features and initial landmark sequence features in Step 1 are converted into enhanced visual features and enhanced landmark features, respectively.

[0010] Step 3 specifically involves: Step 3.1: Gating Vector Calculation: Input the initial audio features generated in Step 1 into the spectrum gating, perform a linear transformation using the learnable weight matrix and bias term, and then normalize the values ​​to between zero and one using the Sigmoid activation function to generate a frequency-sensitive weight mask. Step 3.2: Feature Selective Enhancement: Perform Hadamard product element-wise multiplication on the initial audio features generated in Step 1 and the weight mask generated in Step 3.1 to output the enhanced audio features after noise reduction and enhancement.

[0011] Step 4 specifically involves: Step 4.1: Cross-attention matrix calculation and weighting: The enhanced landmark features output from Step 2.3 are used to generate a guided query vector through linear projection. The enhanced visual features output from Step 2.3 are used to generate key and value vectors through linear projection. The query vector and the transpose of the key vector are multiplied by a dot product and divided by a dimension scaling factor to prevent gradient vanishing. Then, the spatial region attention weight matrix is ​​obtained by normalization using the Softmax function. Finally, the value vectors are weighted and summed using the weight matrix to calculate the spatial attention features guided by landmarks.

[0012] Step 4.2: Feature Residual Fusion and Normalization: The spatial attention features guided by landmarks obtained in Step 4.1 are added element-wise with the enhanced visual features obtained in Step 2.3 after linear projection transformation. The result is then fed into a normalization layer for feature smoothing, and finally the reconstructed visual features are output.

[0013] Step 5 specifically involves: Step 5.1: Obtain the corresponding features by performing three-way parallel mutual attention calculation; Step 5.2: Three-way feature splicing and fusion.

[0014] Step 5.1 specifically involves: First channel: The query vector is generated by projecting the enhanced audio features output in step 3.2, and the key vector and value vector are generated by projecting the reconstructed visual features output in step 4.2. The voice-guided visual interaction features are calculated through a multi-head attention mechanism. The second channel: the reconstructed visual features output in step 4.2 are projected to generate a query vector, and the enhanced audio features output in step 3.2 are projected to generate a key vector and a value vector. The visually guided voice interaction features are calculated through a multi-head attention mechanism. The third channel: The enhanced audio features output in step 3.2 and the reconstructed visual features output in step 4.2 are concatenated along the sequence dimension to form joint features. The joint query vector, joint key vector and joint value vector are generated by projecting them onto the joint features. The cross-modal joint self-attention features are calculated through a multi-head attention mechanism.

[0015] Step 5.2 specifically involves splicing and fusing the visual interaction features output from the first channel, the voice interaction features output from the second channel, and the cross-modal joint self-attention features output from the third channel along the feature channel dimension to generate a high-order fusion representation vector.

[0016] Step 6 specifically involves: Step 6.1: One-dimensional convolutional spatial dimension enhancement and normalization: Input the high-order fusion representation vector output from step 5.2 into a one-dimensional convolutional layer to extract local spatial correlation features between different channels. Then, stabilize the feature numerical distribution through the layer normalization operator to obtain spatially enhanced feature representation. Step 6.2: Global Pooling and Probabilistic Decision Output: The spatial augmentation feature representation obtained in Step 6.1 is compressed in time dimension using the global average pooling operator to extract a global feature vector of fixed length. Then, the Softmax normalized exponential activation function is used to calculate and output the classification prediction label of whether the current single subject belongs to the patient with depression or the non-depression population.

[0017] Step 6 specifically involves: Step 7 specifically involves repeating steps 1 to 6 to obtain the classification prediction labels for all subject samples in the test set; comparing the prediction labels of all subjects with the clinical real labels one by one, counting the total number of correctly predicted samples, and calculating the classification prediction accuracy of the model on the entire dataset using the following formula.

[0018] The beneficial effects of this invention are: (1) The method of this invention solves the shortcomings of cross-modal mining and realizes deep semantic fusion: In view of the shortcomings of simple splicing in capturing the complementary relationship between audio and video, a three-way parallel mutual attention (Mutual Transformer) processing flow is used. Through bidirectional mutual query of speech and vision and joint self-attention, time asynchrony and semantic misalignment are eliminated, and accurate alignment and mining of deep complementary semantics are realized.

[0019] (2) The method of the present invention overcomes the weakness in capturing subtle features and improves the accuracy of micro-feature perception: In response to the problem that depressive features are hidden in facial micro-expressions and specific audio segments, the method combines spatiotemporal frequency domain joint mining (TDFM) and landmark guidance (LGM). TDFM extracts dynamic trajectories and frequency domain micro-tremors, while LGM guides visual focus on areas with significant emotions such as the eyes and mouth, thus avoiding feature loss.

[0020] (3) The method of the present invention has stronger noise resistance: eliminating noise and irrelevant interference, significantly enhancing detection robustness: In response to the problem that the original data contains a large amount of non-emotional related interference, an adaptive spectrum gate is designed to generate a mask for nonlinear weighting of speech, actively strengthening sensitive frequency bands and suppressing environmental noise, which greatly improves the detection robustness in uncontrolled environments. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall process of the multimodal depression detection method of the present invention; Figure 2This is a schematic diagram of the spatiotemporal frequency domain joint feature mining (TDFM) process in the method of this invention; Figure 3 This is a schematic diagram illustrating the principle of landmark-guided visual semantic reconstruction (LGM) in the method of this invention. Figure 4 This is a flowchart of the three-way parallel mutual attention calculation in the method of the present invention. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0023] This invention provides a multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanisms, such as... Figure 1 As shown, the specific steps include: Step 1: Preprocessing of multimodal perception data for subjects with depression; Step 1 is as follows: Step 1.1: Visual and Landmark Feature Extraction: First, the facial video of the depressed subject is cropped into a sequence of N (N=16) consecutive frames; the global visual feature vector of the image sequence is extracted using a pre-trained image feature extraction backbone network (such as Vision Transformer / ViT or ResNet-50 network) to obtain the visual sequence features; at the same time, the key point coordinates of the image sequence are extracted to obtain the landmark sequence. Step 1.2: Speech signal representation construction: After acquiring a single-channel audio signal that is completely synchronized with the video, perform a short-time Fourier transform and map the linear spectrum from high frequency to low frequency onto a group of M (M=80) channels of Mel filters to extract a Mel spectrogram sequence with a size of M times the number of audio time steps. Step 1.3: Unified Representation Space Mapping: Map the visual sequence, landmark sequence, and Mel spectrogram sequence extracted in Step 1.1 to the same latent feature dimension to generate initial visual sequence features, initial landmark sequence features, and initial audio features in a unified feature space.

[0024] Step 2, Spatiotemporal Frequency Domain Joint Feature Mining (TDFM), such as Figure 2 As shown; Step 2 is as follows: Step 2.1: Temporal branch feature extraction: Input the visual sequence features and landmark sequence features extracted in Step 1.1 into a bidirectional gated recurrent unit (Bi-GRU). Establish long-range temporal dependencies by forward and backward recursive calculations along the time axis. Concatenate the forward hidden state vector and the backward hidden state vector, and calculate the mean along the time step to obtain the temporal feature representation of the macroscopic action trajectory. Step 2.2: Frequency domain branch feature extraction: The initial visual sequence features and initial landmark sequence features obtained in Step 1.3 are converted into complex spectrum matrices in the frequency domain using real-valued fast Fourier transform (RFFT). The modulus of the complex spectrum matrix is ​​taken to obtain the spectrum amplitude map, and then the distribution is normalized and adjusted to the same feature dimension as the time domain branch output features in Step 2.1 through a linear projection layer to obtain the frequency domain feature representation. Step 2.3: Dynamic Residual Fusion: The temporal feature representation obtained in Step 2.1 is added and fused with the frequency domain feature representation (obtained in Step 2.2) multiplied by a preset weight coefficient (e.g., set to 0.1). Then, the feature distribution is smoothed by Layer Normalization (LayerNorm) to output the spatiotemporal frequency domain enhanced feature representation, which includes enhanced visual features and enhanced landmark features. Through this step, the initial visual sequence features and initial landmark sequence features in Step 1 are converted into enhanced visual features and enhanced landmark features, respectively.

[0025] Step 3: Spectrum-gated guided speech enhancement; Step 3 specifically involves: Step 3.1: Gating Vector Calculation: Input the initial audio features generated in Step 1 into the spectrum gating, perform a linear transformation using the learnable weight matrix and bias term, and then normalize the values ​​to between zero and one using the Sigmoid activation function to generate a frequency-sensitive weight mask. Step 3.2: Feature Selective Enhancement: Perform Hadamard product element-wise multiplication on the initial audio features generated in Step 1 and the weight mask generated in Step 3.1 to selectively enhance the depression-sensitive frequency band and suppress background noise, and output the enhanced audio features after noise reduction and enhancement.

[0026] Step 4, Landmark-guided visual semantic reconstruction (LGM), such as Figure 3 As shown: Step 4 specifically involves: Step 4.1: Cross-attention matrix calculation and weighting: The enhanced landmark features output from Step 2.3 are used to generate a guided query vector (Query) through linear projection. The enhanced visual features output from Step 2.3 are used to generate a key vector (Key) and a value vector (Value) through linear projection. The query vector and the transpose of the key vector are multiplied by a dot product and divided by a dimension scaling factor to prevent gradient vanishing. Then, the matrix is ​​normalized using the Softmax function to obtain the spatial region attention weight matrix. Finally, the value vector is weighted and summed using the weight matrix to calculate the spatial attention features guided by landmarks.

[0027] Step 4.2: Feature Residual Fusion and Normalization: The spatial attention features guided by landmarks obtained in Step 4.1 are added element-wise with the enhanced visual features obtained in Step 2.3 after linear projection transformation. The result is then fed into a normalization layer for feature smoothing, and finally the reconstructed visual features are output.

[0028] Step 5: Three-way parallel mutual attention cross-modal fusion (Mutual Transformer), such as... Figure 4 As shown; Step 5 specifically involves: Step 5.1: Three-way parallel mutual attention calculation, specifically: The first channel (voice query vision, MT-1): the query vector is generated by projecting the enhanced audio features output in step 3.2, the key vector and value vector are generated by projecting the reconstructed visual features output in step 4.2, and the voice-guided visual interaction features are calculated through a multi-head attention mechanism. The second channel (visual query speech, MT-2): the query vector is generated by projecting the reconstructed visual features output in step 4.2, and the key vector and value vector are generated by projecting the enhanced audio features output in step 3.2. The visually guided speech interaction features are calculated through a multi-head attention mechanism. The third channel (audio-video joint self-enhancement, MT-3): The enhanced audio features output in step 3.2 and the reconstructed visual features output in step 4.2 are concatenated along the sequence dimension to form joint features. The joint features are projected to generate joint query vector, joint key vector and joint value vector. Cross-modal joint self-attention features are calculated through a multi-head attention mechanism. Step 5.2: Multi-channel feature splicing and fusion: The visual interaction features output from the first channel, the voice interaction features output from the second channel, and the cross-modal joint self-attention features output from the third channel are spliced ​​and fused along the feature channel dimension to generate a high-order fusion representation vector.

[0029] Step 6: Spatial Augmentation and Classification: Step 6 specifically involves: Step 6.1: One-dimensional convolutional spatial dimension enhancement and normalization: Input the high-order fusion representation vector output from step 5.2 into a one-dimensional convolutional layer to extract local spatial correlation features between different channels. Then, stabilize the feature numerical distribution through the layer normalization operator to obtain spatially enhanced feature representation. Step 6.2: Global Pooling and Probabilistic Decision Output: The spatial augmentation feature representation obtained in Step 6.1 is compressed in time dimension using the global average pooling operator to extract a global feature vector of fixed length. Then, the Softmax normalized exponential activation function is used to calculate and output the classification prediction label of whether the current single subject belongs to the patient with depression or the non-depression population.

[0030] Step 7: Decision statistics for the sample set and evaluation of global indicators; Step 7 specifically involves repeating steps 1 to 6 to sequentially obtain the classification prediction labels for all subject samples in the test set; comparing the prediction labels of all subjects with the clinical real labels one by one, counting the total number of correctly predicted samples, and calculating the classification prediction accuracy (Acc) of the model on the entire dataset using the following formula.

[0031] Example 1 1. Experimental Dataset and Parameter Configuration Test dataset: The internationally recognized depression detection dataset AVEC2014 was used. This dataset contains facial videos and synchronized speech signals of subjects in an uncontrolled environment.

[0032] 1.1 Data Preprocessing and Feature Dimension Setting (corresponding to step 1): Visual and Landmark Analysis: Facial videos were cropped into a sequence of 16 consecutive frames. Global visual features were extracted using a pre-trained ViT / ResNet-50 algorithm with an output dimension of 16×136. OpenFace was used to extract 68 keypoints (136-dimensional coordinates) per frame, with an output dimension of 16×136.

[0033] Speech signal: The single-channel audio sampling rate is 16,000 Hz, the window length is 25 ms, the frame shift is 10 ms, and it is mapped to an 80-channel Mel filter via STFT to extract a Mel spectrogram with a size of Ta×80 (Ta is the audio time step).

[0034] Dimension mapping: The initial visual sequence features (16×512), initial landmark sequence features (16×512), and initial audio features (Ta×512) are generated by mapping to a unified latent dimension d = 512 through a linear projection layer.

[0035] 1.2 Setting core module structural parameters (corresponding to steps 2-6): Spatiotemporal Frequency Joint Feature Mining (TDFM): The Bi-GRU hidden layer dimension is set to 256 (512 after bidirectional stitching); RFFT calculates the complex spectrum along the time axis (16 frames) and takes the modulus, the linear projection layer is adjusted to 512 dimensions, and after LayerNorm, the output is enhanced visual features (16×512) and enhanced landmark features (16×512).

[0036] Spectrum-gated guided speech enhancement: The weight dimension of the fully connected layer is 512×512. After Sigmoid mapping, it is multiplied by the initial audio features element-wise to output the enhanced audio features (Ta×512).

[0037] Landmark-guided visual semantic reconstruction (LGM): Q, K, and V linear projection layers all have a dimension of 512, and the attention scaling factor is [value missing]. The visual features (16×512) are reconstructed from the residual and LayerNorm normalized output.

[0038] Three-way parallel mutual attention computation: The three channels (MT-1, MT-2, MT-3) all adopt an 8-head cross / self-attention mechanism (64 dimensions per head, 512 dimensions in total). After temporal mean pooling, each of the three channels outputs a 1×512-dimensional vector, and the ends are concatenated to generate a 1×1536-dimensional high-order fusion representation vector.

[0039] Spatial augmentation and classification: One-dimensional convolution (Conv1d) with kernel size k=3, stride s=1, and number of channels 1536; after global average pooling (GAP), it is input into a 2-layer multilayer perceptron, and finally outputs 2-dimensional classification prediction probability through Softmax.

[0040] 1.3 Model training hyperparameters: The AdamW optimizer was used, with a learning rate of 0.00001, a batch size of 16, a dropout probability of 0.1, and a cross-entropy loss function to guide gradient updates.

[0041] 2. Progressive Comparative Verification Process and Experimental Result Analysis Option 1: Remove spectrum gating (remove step 3): Based on the complete method (MMFformer_Adapted), step 3 is removed. That is, the initial audio features are directly fed into the subsequent cross-modal interaction module instead of using frequency-sensitive weighted masks for dimensionality reduction and noise filtering. This scheme is abbreviated as MMFformer + TDFM + LGM + MT.

[0042] Option 2: Remove spectrum gating, landmark guidance, and mutual attention network (remove steps 3, 4, and 5): Based on Scheme 1, steps 4 and 5 are further removed, retaining only step 2 for extracting enhanced features. Then, the audio and video features are directly subjected to time averaging and simple matrix concatenation. This scheme is abbreviated as MMFformer + TDFM.

[0043] Option 3: Remove all core enhancement modules / baseline solutions (remove steps 2, 3, 4, and 5): Based on Scheme 2, step 2 is further removed, meaning that spatiotemporal frequency domain joint feature extraction and spectral gating enhancement are completely omitted. Instead, only basic visual and audio Transformers are used to extract features, followed by simple concatenation and classification decisions. This scheme is abbreviated as MMFformer (baseline).

[0044] Option 4: The complete solution of the present invention (including steps 1 to 6): The complete structure of this invention (MMFformer_Adapted) is included, and adaptive spectrum gating, TDFM module, landmark guidance module LGM and three-way parallel mutual attention network MT are introduced to achieve deep enhancement and collaborative discrimination of multimodal information.

[0045] Evaluation Results and Performance Comparison: On the AVEC2014 dataset, the evaluation was conducted using metrics such as accuracy, F1 score, AUC (area under the ROC curve), precision, and recall. Typical comparative test results are shown in the table below: Table 1 shows the performance of Freeform on MMFformer_Adapted in the AVEC2014 dataset.

[0046] As shown in the table above, adding TDFM improved the accuracy from 72.00% to 74.00%, the F1-Score by 3.28% (to 76.36%), and the precision by 6.16% (to 75.86%), indicating that extracting frequency domain energy and time domain trajectory can accurately capture subtle facial tremors and micro-expressions in depression. Adding LGM and MT further improved the accuracy by 4.00% (to 78.00%), proving that landmark-guided visual semantic reconstruction combined with three-way parallel mutual attention calculation successfully eliminated temporal asynchrony and semantic misalignment between audio and video modalities. Compared to the scheme without spectrum gating, the complete method of this invention improved the accuracy to 80.00%, the AUC to 82.43%, and the precision to 95.24% (a significant improvement of 22.16%), greatly reducing the risk of misdiagnosis in healthy individuals.

[0047] Example 2 A multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanisms includes the following steps: Step 1: Preprocessing of multimodal perception data for subjects with depression; Step 2: Joint feature mining in the spatiotemporal and frequency domains; Step 3: Spectrum-gated guided speech enhancement; Step 4: Visual semantic reconstruction guided by landmarks; Step 5: Three-way parallel mutual attention cross-modal fusion; Step 6: Spatial augmentation and classification; Step 7: Decision statistics of sample set and evaluation of global indicators.

[0048] Example 3 A multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanisms includes the following steps: Step 1: Preprocessing of multimodal perception data for subjects with depression; Step 1 is as follows: Step 1.1: Visual and Landmark Feature Extraction: First, the facial video of the depressed subject is cropped into a sequence of N consecutive frames; the global visual feature vector of the image sequence is extracted using a pre-trained image feature extraction backbone network to obtain the visual sequence features; at the same time, the key point coordinates of the image sequence are extracted to obtain the landmark sequence. Step 1.2: Speech signal representation construction: After acquiring a single-channel audio signal that is completely synchronized with the video, perform a short-time Fourier transform and map the linear spectrum from high frequency to low frequency onto an M-channel Mel filter group to extract a Mel spectrogram sequence with a size equal to the number of audio time steps multiplied by M dimensions. Step 1.3: Unified Representation Space Mapping: Map the visual sequence, landmark sequence, and Mel spectrogram sequence extracted in Step 1.1 to the same latent feature dimension to generate initial visual sequence features, initial landmark sequence features, and initial audio features in a unified feature space.

[0049] Step 2: Joint feature mining in the spatiotemporal and frequency domains; Step 3: Spectrum-gated guided speech enhancement; Step 4: Visual semantic reconstruction guided by landmarks; Step 5: Three-way parallel mutual attention cross-modal fusion; Step 6: Spatial augmentation and classification; Step 7: Decision statistics of sample set and evaluation of global indicators.

[0050] Example 4 A multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanisms includes the following steps: Step 1: Preprocessing of multimodal perception data for subjects with depression; Step 2: Joint feature mining in the spatiotemporal and frequency domains; Step 2 is as follows: Step 2.1: Temporal branch feature extraction: Input the visual sequence features and landmark sequence features extracted in Step 1.1 into the bidirectional gated recurrent unit, establish long-range temporal dependencies by forward and backward recursive calculations along the time axis, concatenate the forward hidden state vector and the backward hidden state vector, and calculate the mean along the time step to obtain the temporal feature representation of the macroscopic action trajectory. Step 2.2: Frequency domain branch feature extraction: The initial visual sequence features and initial landmark sequence features obtained in Step 1.3 are converted into complex spectrum matrices in the frequency domain using real-valued fast Fourier transform. The modulus of the complex spectrum matrix is ​​taken to obtain the spectrum amplitude map, and then the distribution is normalized and adjusted to the same feature dimension as the time domain branch output features in Step 2.1 through a linear projection layer to obtain the frequency domain feature representation. Step 2.3: Dynamic Residual Fusion: The temporal feature representation obtained in Step 2.1 is added and fused with the frequency domain feature representation multiplied by a preset weight coefficient. Then, the feature distribution is smoothed by layer normalization to output the spatiotemporal frequency domain enhanced feature representation, which includes enhanced visual features and enhanced landmark features. Through this step, the initial visual sequence features and initial landmark sequence features in Step 1 are converted into enhanced visual features and enhanced landmark features, respectively.

[0051] Step 3: Spectrum-gated guided speech enhancement; Step 4: Visual semantic reconstruction guided by landmarks; Step 5: Three-way parallel mutual attention cross-modal fusion; Step 6: Spatial augmentation and classification; Step 7: Decision statistics of sample set and evaluation of global indicators.

[0052] Example 5 A multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanisms includes the following steps: Step 1: Preprocessing of multimodal perception data for subjects with depression; Step 2: Joint feature mining in the spatiotemporal and frequency domains; Step 3: Spectrum-gated guided speech enhancement; Step 3 specifically involves: Step 3.1: Gating Vector Calculation: Input the initial audio features generated in Step 1 into the spectrum gating, perform a linear transformation using the learnable weight matrix and bias term, and then normalize the values ​​to between zero and one using the Sigmoid activation function to generate a frequency-sensitive weight mask. Step 3.2: Feature Selective Enhancement: Perform Hadamard product element-wise multiplication on the initial audio features generated in Step 1 and the weight mask generated in Step 3.1 to output the enhanced audio features after noise reduction and enhancement.

[0053] Step 4: Visual semantic reconstruction guided by landmarks; Step 5: Three-way parallel mutual attention cross-modal fusion; Step 6: Spatial augmentation and classification; Step 7: Decision statistics of sample set and evaluation of global indicators.

[0054] Example 6 A multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanisms includes the following steps: Step 1: Preprocessing of multimodal perception data for subjects with depression; Step 2: Joint feature mining in the spatiotemporal and frequency domains; Step 3: Spectrum-gated guided speech enhancement; Step 4: Visual semantic reconstruction guided by landmarks; Step 4 specifically involves: Step 4.1: Cross-attention matrix calculation and weighting: The enhanced landmark features output from Step 2.3 are used to generate a guided query vector through linear projection. The enhanced visual features output from Step 2.3 are used to generate key and value vectors through linear projection. The query vector and the transpose of the key vector are multiplied by a dot product and divided by a dimension scaling factor to prevent gradient vanishing. Then, the spatial region attention weight matrix is ​​obtained by normalization using the Softmax function. Finally, the value vectors are weighted and summed using the weight matrix to calculate the spatial attention features guided by landmarks.

[0055] Step 4.2: Feature Residual Fusion and Normalization: The spatial attention features guided by landmarks obtained in Step 4.1 are added element-wise with the enhanced visual features obtained in Step 2.3 after linear projection transformation. The result is then fed into a normalization layer for feature smoothing, and finally the reconstructed visual features are output.

[0056] Step 5: Three-way parallel mutual attention cross-modal fusion; Step 6: Spatial augmentation and classification; Step 7: Decision statistics of sample set and evaluation of global indicators.

Claims

1. A multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanism, characterized in that, Specifically, the following steps are included: Step 1: Preprocessing of multimodal perception data for subjects with depression; Step 2: Joint feature mining in the spatiotemporal and frequency domains; Step 3: Spectrum-gated guided speech enhancement; Step 4: Visual semantic reconstruction guided by landmarks; Step 5: Three-way parallel mutual attention cross-modal fusion; Step 6: Spatial augmentation and classification; Step 7: Decision statistics of sample set and evaluation of global indicators.

2. The multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanism according to claim 1, characterized in that, Step 1 is as follows: Step 1.1: Visual and Landmark Feature Extraction: First, the facial video of the depressed subject is cropped into a sequence of N consecutive frames; the global visual feature vector of the image sequence is extracted using a pre-trained image feature extraction backbone network to obtain the visual sequence features; at the same time, the key point coordinates of the image sequence are extracted to obtain the landmark sequence. Step 1.2: Speech signal representation construction: After acquiring a single-channel audio signal that is completely synchronized with the video, perform a short-time Fourier transform and map the linear spectrum from high frequency to low frequency onto an M-channel Mel filter group to extract a Mel spectrogram sequence with a size equal to the number of audio time steps multiplied by M dimensions. Step 1.3: Unified Representation Space Mapping: Map the visual sequence, landmark sequence, and Mel spectrogram sequence extracted in Step 1.1 to the same latent feature dimension to generate initial visual sequence features, initial landmark sequence features, and initial audio features in a unified feature space.

3. The multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanism according to claim 2, characterized in that, Step 2 is as follows: Step 2.1: Temporal branch feature extraction: Input the visual sequence features and landmark sequence features extracted in Step 1.1 into the bidirectional gated recurrent unit, establish long-range temporal dependencies by forward and backward recursive calculations along the time axis, concatenate the forward hidden state vector and the backward hidden state vector, and calculate the mean along the time step to obtain the temporal feature representation of the macroscopic action trajectory. Step 2.2: Frequency domain branch feature extraction: The initial visual sequence features and initial landmark sequence features obtained in Step 1.3 are converted into complex spectrum matrices in the frequency domain using real-valued fast Fourier transform. The modulus of the complex spectrum matrix is ​​taken to obtain the spectrum amplitude map, and then the distribution is normalized and adjusted to the same feature dimension as the time domain branch output features in Step 2.1 through a linear projection layer to obtain the frequency domain feature representation. Step 2.3: Dynamic residual fusion: The temporal feature representation obtained in Step 2.1 is added and fused with the frequency domain feature representation multiplied by a preset weight coefficient. Then, the feature distribution is smoothed by layer normalization to output the spatiotemporal frequency domain enhanced feature representation, which includes enhanced visual features and enhanced landmark features. Through this step, the initial visual sequence features and initial landmark sequence features in step 1 are converted into enhanced visual features and enhanced landmark features, respectively.

4. The multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanism according to claim 3, characterized in that, Step 3 specifically involves: Step 3.1: Gating Vector Calculation: Input the initial audio features generated in Step 1 into the spectrum gating, perform a linear transformation using the learnable weight matrix and bias term, and then normalize the values ​​to between zero and one using the Sigmoid activation function to generate a frequency-sensitive weight mask. Step 3.2: Feature Selective Enhancement: Perform Hadamard product element-wise multiplication on the initial audio features generated in Step 1 and the weight mask generated in Step 3.1 to output the enhanced audio features after noise reduction and enhancement.

5. The multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanism according to claim 4, characterized in that, Step 4 specifically involves: Step 4.1: Cross-Attention Matrix Calculation and Weighting: The enhanced landmark features output from Step 2.3 are used to generate a guided query vector through linear projection. The enhanced visual features output from Step 2.3 are used to generate key and value vectors through linear projection. The query vector and the transpose of the key vector are multiplied by a dot product and divided by a dimension scaling factor to prevent gradient vanishing. Then, the matrix is ​​normalized using the Softmax function to obtain the spatial region attention weight matrix. Finally, the value vectors are weighted and summed using the weight matrix to calculate the spatial attention features guided by landmarks. Step 4.2: Feature Residual Fusion and Normalization: The spatial attention features guided by landmarks obtained in Step 4.1 are added element-wise with the enhanced visual features obtained in Step 2.3 after linear projection transformation. The result is then fed into a normalization layer for feature smoothing, and finally the reconstructed visual features are output.

6. The multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanism according to claim 5, characterized in that, Step 5 specifically involves: Step 5.1: Obtain the corresponding features by performing three-way parallel mutual attention calculation; Step 5.2: Three-way feature splicing and fusion.

7. The multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanism according to claim 6, characterized in that, Step 5.1 specifically involves: First channel: The query vector is generated by projecting the enhanced audio features output in step 3.2, and the key vector and value vector are generated by projecting the reconstructed visual features output in step 4.

2. The voice-guided visual interaction features are calculated through a multi-head attention mechanism. The second channel: the reconstructed visual features output in step 4.2 are projected to generate a query vector, and the enhanced audio features output in step 3.2 are projected to generate a key vector and a value vector. The visually guided voice interaction features are calculated through a multi-head attention mechanism. The third channel: The enhanced audio features output in step 3.2 and the reconstructed visual features output in step 4.2 are concatenated along the sequence dimension to form joint features. The joint query vector, joint key vector and joint value vector are generated by projecting them onto the joint features. The cross-modal joint self-attention features are calculated through a multi-head attention mechanism.

8. The multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanism according to claim 7, characterized in that, Step 5.2 specifically involves splicing and fusing the visual interaction features output from the first channel, the voice interaction features output from the second channel, and the cross-modal joint self-attention features output from the third channel along the feature channel dimension to generate a high-order fusion representation vector.

9. The multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanism according to claim 8, characterized in that, Step 6 specifically involves: Step 6.1: One-dimensional convolutional spatial dimension enhancement and normalization: Input the high-order fusion representation vector output from step 5.2 into a one-dimensional convolutional layer to extract local spatial correlation features between different channels. Then, stabilize the feature numerical distribution through the layer normalization operator to obtain spatially enhanced feature representation. Step 6.2: Global Pooling and Probabilistic Decision Output: The spatial augmentation feature representation obtained in Step 6.1 is compressed in time dimension using the global average pooling operator to extract a global feature vector of fixed length. Then, the Softmax normalized exponential activation function is used to calculate and output the classification prediction label of whether the current single subject belongs to the patient with depression or the non-depression population.

10. The multimodal depression detection method based on spatiotemporal frequency domain enhancement and mutual attention mechanism according to claim 9, characterized in that, Step 6 specifically involves: Step 7 specifically involves repeating steps 1 to 6 to obtain the classification prediction labels for all subject samples in the test set; comparing the prediction labels of all subjects with the clinical real labels one by one, counting the total number of correctly predicted samples, and calculating the classification prediction accuracy of the model on the entire dataset using the following formula.

Images