Depression detection method fused with multi-modal attention mechanism

Abstract

The invention discloses a depression detection method fusing a multi-modal attention mechanism, and relates to the technical field of auxiliary psychological health diagnosis, and the method comprises the steps: respectively extracting a facial image and a voice signal from an original video, extracting a global feature map from the facial image through a deep convolutional network, and introducing a local fusion module and a global fusion module; combining a Gaussian pyramid attention module with a spatial domain attention module; speech signals are converted into a Mel spectrogram through preprocessing, speech features are extracted through a SincNet network, and the speech features are further sent to a time-frequency attention module to highlight depression-related intonation and energy changes; in the feature fusion stage, a self-adaptive attention mechanism is adopted, self-attention modeling is performed on face and voice modes, cross-mode fusion and context modeling are completed through a Query-Key-Value structure, and a depression score is output through a full connection layer. According to the scheme, the problem that cross-modal nonlinear correlation is difficult to dynamically capture is solved, and effective technical support is provided for mental health assessment and intervention.

Description

Technical Field

[0001] This invention is based on the field of deep learning-assisted mental health diagnosis technology, specifically a depression detection method that integrates multimodal attention mechanisms. By integrating facial micro-expressions and voice features in deep analysis, it achieves an objective quantitative assessment of the degree of depression. Background Technology

[0002] With the increasing application of multimodal data in the field of depression identification, existing detection methods still face significant challenges in cross-modal collaborative modeling.

[0003] Current technical approaches have the following core limitations: traditional methods rely heavily on single-modal data and fail to effectively integrate the complementary advantages of multi-source depression representations; existing multimodal fusion strategies struggle to dynamically capture nonlinear relationships between modalities, resulting in coarse-grained feature interactions; key depression-sensitive information is at risk of deep loss—the transient characteristics of facial micro-expressions and the nonlinear dynamic features of speech prosody are not adequately modeled; and in particular, there is a lack of adaptive cross-modal interaction mechanisms for depression-sensitive features, leading to insufficient sensitivity of the model to subtle changes in emotion.

[0004] Therefore, it is necessary to construct a collaborative perception framework that can deeply mine multimodal complementary characteristics, dynamically fuse cross-modal associations, and adaptively enhance the expression of depressive sensitivity features. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for detecting depression that integrates multimodal attention mechanisms. To achieve the above objectives, the technical solution adopted by this invention is as follows: A method for detecting depression that integrates multimodal attention mechanisms includes: S1: Process the original video to extract facial images and speech signals respectively; the facial image sequence is used as the input of the facial extraction network branch, and the speech signal is used as the input of the speech extraction network. S2: Use a convolutional network to extract features from facial images to obtain global feature maps of the face, providing complete visual emotion information; input the extracted global feature maps into the local and global fusion module to generate fused feature maps containing contextual information, and then pass through the Gaussian pyramid module and the spatial domain attention module to obtain multi-scale facial feature maps containing key regions. S3: For the audio branch, the speech signal is preprocessed using speech features, converted into a Mel spectrogram, and then input into a convolutional network to extract audio features. Subsequently, it is input into the time-frequency attention module to enhance the expression of features related to tone, speech rate and depression from both time and frequency domains. S4: Multimodal features from the facial image pathway and the speech pathway are input into the multimodal adaptive attention module. The intramodal relationship is explored through self-attention mechanism to achieve dynamic fusion of cross-modal features. The weights are adjusted through an adaptive control gate to integrate speech and visual emotion information. S5: The fused multimodal feature map is expanded by global average pooling and then input into the fully connected layer, finally outputting a depression detection score to achieve a quantitative assessment of the emotional state of the tested person.

[0006] Preferably, S1 specifically includes: By preprocessing the original video, facial image sequences and speech signals are separated and extracted. The facial image sequences are input into a facial extraction network to extract multi-scale expression features of key facial regions, and the speech signals are input into a speech feature network to capture emotional and speech feature information in speech patterns.

[0007] Preferably, S2 specifically includes: The facial image is first processed by a deep convolutional network ResNet18 to extract global feature maps. Then, it is input into a local and global fusion module to enhance the ability to express contextual features. It then passes through a Gaussian pyramid module and a spatial domain attention module in sequence to finally obtain a multi-scale facial feature map containing information about key regions.

[0008] Preferably, S3 specifically includes: After performing Mel spectrogram transformation on the speech signal, it is input into the SincNet network to extract audio features, and then input into the time-frequency attention module to enhance the expression of features related to tone, speech rate and depression from both time and frequency domains.

[0009] Preferably, S4 specifically includes: The multimodal adaptive attention module models the intramodal relationship between facial and speech through a self-attention mechanism and realizes dynamic fusion of intermodal features based on a query-key-value structure. The fusion process is controlled by an adaptive control gate to adjust the weight of each modal information, thereby integrating the emotional features of speech and vision.

[0010] Preferably, S,5 specifically includes: The prediction module performs global average pooling on the fused multimodal feature map, flattens it into a fixed-dimensional feature vector, and outputs a depression score through a fully connected layer to achieve a quantitative assessment of the subject's emotional state.

[0011] Preferably, the dual-modal detection model specifically includes: We employ parallel visual and speech feature extraction pathways and achieve dynamic fusion between modalities through a multimodal adaptive attention module. The model input is raw video data, and the output is quantitative scoring results related to depression.

[0012] Preferably, the processing flow of the PGF module, Gaussian pyramid module, and spatial domain attention module in S2 is as follows: S21: Input the global facial feature map extracted by ResNet18 into the PGF module. By integrating the local and global information through convolutional fusion of the cropped local feature map and the global feature map, the fused feature map containing the context is output. S22: Input the PGF output into the Gaussian pyramid module, generate pyramid feature maps of different scales through multi-layer downsampling, then perform upsampling reconstruction in sequence, and obtain the corresponding scale attention map through the Sigmoid function; S23: After weighted fusion of attention maps at each scale, multiply them element-wise with the original fused feature map to generate the final Gaussian pyramid attention feature map; S24: Input the above attention feature map into the spatial domain attention module, extract the attention weights of key regions in the spatial domain through depthwise separable convolution and standard convolution operations, and finally weight and fuse them with the fusion feature map to obtain a multi-scale spatial attention feature map containing key facial regions.

[0013] Preferably, the subsequent process of speech feature processing in S3 is as follows: S31: Preprocess the acquired speech signal, including noise reduction, endpoint detection and framing, and extract Mel spectrogram as speech input features; S32: Input the Mel spectrogram into the SincNet network to extract speech feature maps containing intonation, speech rate, and time-domain and frequency-domain related features; S33: Input the extracted speech feature map into the time-frequency attention module, use the idea of ​​residuals to connect the two attentions to enhance the time and frequency dimension information of the speech, and generate a time-frequency weighted map through the Sigmoid function; S34: Element-wise weighting of the time-frequency weighted map and the original speech feature map is performed to obtain the final speech attention feature map for subsequent fusion.

[0014] Preferably, the process of the multimodal adaptive attention fusion module in S4 is as follows: S41: The facial feature map obtained by fusing Gaussian pyramid and spatial domain attention, and the speech feature map enhanced by time-frequency attention, are respectively input into the self-attention mechanism pathway to capture long-term intramodal dependencies; S42: Use the Query-Key-Value structure to construct a cross-modal attention path to achieve dynamic matching and complementary representation between speech and visual information; S43: The adaptive control gate mechanism is used to assign dynamic weight coefficients to the two modalities and fuse them to generate a unified multimodal emotion feature map; S44: Integrate cross-modal attention. The query comes from the main modality, namely facial feature self-attention, while the key and value come from speech feature self-attention. S45: Finally, the visual self-attention and the visual-guided speech attention are added together to obtain the final cross-modal features.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention comprehensively utilizes two data sources: facial images and speech signals. By constructing a Gaussian pyramid attention module and a spatial domain attention module for facial images, it effectively extracts multi-scale and local key region features of the face, enhancing the model's ability to perceive facial expression details and structural information. Meanwhile, for speech signals, a time-frequency joint attention mechanism is introduced to fully capture emotional cues in speech in the time and frequency dimensions.

[0016] Furthermore, this invention designs a multimodal adaptive fusion module that employs a cross-modal attention mechanism and a dynamic weight allocation strategy to achieve deep complementarity and fusion of visual and audio information.

[0017] Through the synergistic effect of the above structures, this invention can identify depression-related behavioral features from multiple perspectives and across modalities, significantly improving the discriminative ability of the detection model. Therefore, this invention has significant advantages such as high detection accuracy, comprehensive feature representation, and strong cross-modal adaptability, making it suitable for various application scenarios such as clinical auxiliary diagnosis and mental health assessment. Attached Figure Description

[0018] Figure 1 This is an architecture diagram of a depression detection method that integrates multimodal attention mechanisms; Figure 2 This is a diagram of the time-frequency attention architecture of the present invention; Figure 3 This is a diagram of the multimodal adaptive attention architecture of the present invention. Detailed Implementation

[0019] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0020] Reference Figure 1 As shown, this invention provides a method for detecting depression that integrates multimodal attention mechanisms, comprising the following steps: (1) Extract facial images and speech signals from the original video; (2) Perform feature extraction and emotion modeling on the facial images; (3) Perform feature extraction and emotion modeling on the speech signals; (4) Use multimodal adaptive attention to fuse facial and speech features; (5) After fusing multimodal features, input them into a fully connected layer and finally output a depression detection score.

[0021] In step (1), the original video can be RGB video or video data in other formats. Understandably, the extracted facial images and speech signals need to undergo certain preprocessing to adapt them to the input of subsequent models. Preprocessing may include operations such as face alignment, speech denoising, and normalization.

[0022] In step (2), a global feature map of the facial image is extracted using a deep convolutional network ResNet18, and a local and global fusion (PGF) module is introduced; further, a Gaussian pyramid attention module and a spatial domain attention module are combined to achieve multi-scale modeling and emotion saliency enhancement of key facial regions. Understandably, the model used in this step is pre-trained, and its training process includes steps such as constructing a facial emotion dataset, initializing model parameters, optimizing the model using training data, and evaluating model performance.

[0023] In step (3), the speech signal is preprocessed into a Mel spectrogram, and speech features are extracted through a SincNet network. These features are then fed into a time-frequency attention module to highlight depression-related intonation and energy changes. Understandably, the speech processing model used in this step also needs to be trained. The training process includes steps such as speech data acquisition, feature extraction, model training, and validation.

[0024] Reference Figure 1 As shown, in a specific embodiment of the present invention, the time-frequency attention module processes the Mel spectrum feature map through S301~S303 to obtain speech features containing attention.

[0025] Step S301: The first branch uses a 1×1 convolution kernel to perform dimensionality reduction on the Mel spectrum feature map, reducing the feature channel dimension. After processing with the ReLU activation function, a non-linear transformation is performed to enhance the feature representation capability. Then, a 1×1 convolution kernel is used again to perform dimensionality restoration. Finally, an attention weight map between 0 and 1 is generated through the Sigmoid function.

[0026] Step S302: In the second processing branch, maintain the integrity of the original Mel-spectral feature map. This branch directly retains the unprocessed original speech feature information.

[0027] Step S303: Multiply the attention weight map generated by the first branch element-wise with the original feature map of the second branch. This multiplication operation achieves adaptive feature enhancement, preserving the integrity of the original features while focusing on strengthening key speech features related to depression detection. The above steps S301~S303 together can be expressed by the following formula: (1) In the formula, The input is the Mel feature spectrum. To reduce the dimensionality to 1 / 4 of the number of channels, 1. For activation function, To recover the original dimensional weight matrix 2, For the Sigmoid function, This is for element-wise multiplication. Bias term. and Used to adjust feature distribution offset.

[0028] In step (4), a multimodal adaptive attention mechanism is used to perform self-attention modeling for facial and speech modalities respectively, and cross-modal fusion and context modeling are completed through a Query-Key-Value structure. Understandably, the training process of this fusion module includes steps such as multimodal data alignment, attention weight learning, and joint optimization.

[0029] Reference Figure 3 As shown, in a specific embodiment of the present invention, the multimodal adaptive attention module performs fusion processing on facial and speech features through S401~S405 to obtain multimodal fused features.

[0030] Step S401: The facial feature map obtained by fusing Gaussian pyramid and spatial domain attention, and the speech feature map enhanced by time-frequency attention, are respectively input into the self-attention mechanism pathway to capture long-term intra-modal dependencies. The modality-specific self-attention coefficients are: (2) In the formula, , , For a learnable projection matrix, Features of the input modality, This is the scaling factor.

[0031] Step S402: Construct a cross-modal attention path using a Query-Key-Value structure to achieve dynamic matching and complementary representation between speech and visual information. The self-attention coefficients for face and speech are: (3) (4) In the formula, It is a facial feature. It is a speech feature.

[0032] Step S403: Dynamic weight coefficients are assigned to the two modalities using an adaptive control gate (ACG) mechanism, and then fused to generate a unified multimodal emotion feature map. The adaptive weights for face and speech are: (5) (6) In the formula, For the sigmoid function, It is a multilayer perceptron. Average pooling, It is a facial feature.

[0033] Step S404: Integrate the cross-modal attention. The Query comes from the main modality, i.e., facial feature self-attention, while the Key and Value come from speech feature self-attention. The cross-modal attention coefficients are: (6) In the formula, , , These are the facial query vector, voice key vector, and voice value vector, respectively. This is the scaling factor.

[0034] Step S405: Finally, the visual self-attention and the visual-guided speech attention are added together to obtain the final cross-modal features.

[0035] The above steps S401~S405 can be expressed by the general formula as follows: (7) In the formula, , These are facial weights and voice weights, respectively. , These are facial and speech self-attention feature maps, respectively. , , It is a learnable projection matrix.

[0036] In step (5), the final depression score is calculated using a fully connected neural network based on the feature vector after multimodal fusion. Specifically, the fused features are first transformed and dimensionality reduced by one or more fully connected layers with nonlinear activation functions. Dropout or batch normalization may be introduced to improve the robustness of the model. Finally, a depression score is generated through a linear output layer, which may be in the form of a continuous value.

[0037] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A depression detection method fusing multi-modal attention mechanism, characterized in that, Comprise: S1: processing the original video, respectively, to extract facial images and speech signals; facial image sequence as the input of facial extraction network branch, speech signal as the input of speech extraction network; S2: the facial image adopts convolution network to extract features, obtains the global feature map of the face, and provides complete visual emotion information; the extracted global feature map is input into the local and global fusion module to generate a fusion feature map containing context information, and then the Gaussian pyramid module and the spatial domain attention module are used to obtain a multi-scale facial feature map containing key regions; S3: the audio branch uses speech features to preprocess the speech signal, converts it into a mel spectrum graph, and then inputs the convolution network to extract audio features, and then inputs the time-frequency attention module to enhance the features related to depression from the time domain and frequency domain; S4: the multi-modal features from the facial image path and the speech path are input into the multi-modal adaptive attention module, the intra-modal relationship is modeled through the self-attention mechanism, the dynamic fusion of cross-modal features is realized, the weight is adjusted through the adaptive control gate, and the speech and visual emotion information is integrated; S5: the fused multi-modal feature map is input into the fully connected layer after global average pooling, and finally the depression detection score is output, realizing the quantitative evaluation of the emotional state of the detected person.

2. The depression detection method of claim 1, wherein, The S1 specifically comprises: By preprocessing the original video, the facial image sequence and the speech signal are separated and extracted, the facial image sequence is input into the facial extraction network to extract the multi-scale expression features of the facial key region, and the speech signal is input into the speech feature network to capture the emotional and speech feature information in the speech mode. 3.The depression detection method of claim 2, wherein, The S2 specifically comprises: The facial image is first extracted by the deep convolution network ResNet18 to obtain the global feature map, and then input into the local and global fusion module to enhance the context feature expression ability, and then sequentially pass through the Gaussian pyramid module and the spatial domain attention module, and finally obtain the multi-scale facial feature map containing key region information.

4. The depression detection method of claim 3, wherein, The S3 specifically comprises: The speech signal is converted into a mel spectrum graph and input into the SincNet network to extract audio features, and then input into the time-frequency attention module to enhance the expression of features related to depression from the time domain and frequency domain.

5. The method of claim 4, wherein the method of detecting depression by fusing multi-modal attention mechanism is characterized by, The S4 specifically comprises: The multi-modal adaptive attention module models the intra-modal relationship of face and speech through the self-attention mechanism, and realizes the dynamic fusion of inter-modal features based on the Query-Key-Value structure. The fusion process is adjusted by the adaptive control gate, and the weight of each modal information is adjusted, so as to integrate the speech and visual emotion features.

6. The depression detection method of claim 5, wherein, The S,5 specifically comprises: The prediction module performs global average pooling on the fused multi-modal feature map, flattens it into a fixed dimension feature vector, and outputs the depression score through the fully connected layer, realizing the quantitative evaluation of the emotional state of the detected person.

7. The method of claim 6, wherein the method of detecting depression by fusing multi-modal attention mechanism is characterized by, The dual-modal detection model specifically comprises: Adopt parallel visual and speech feature extraction channels, and realize dynamic fusion between modalities through a multi-modal adaptive attention module.

8. The method of claim 7, wherein the method of detecting depression by fusing multi-modal attention mechanism is characterized by, The processing flow of the PGF module, the Gaussian pyramid module and the spatial domain attention module in S2 is as follows: S21: input the global facial feature map extracted by ResNet18 into the PGF module, integrate the local and global information by convolution fusion, and output a fusion feature map containing context; S22: input the PGF output into the Gaussian pyramid module, generate pyramid feature maps of different scales through multi-layer downsampling, then sequentially perform upsampling reconstruction, and obtain the corresponding scale attention map through the Sigmoid function; S23: multiply the weighted fusion of the attention maps of each scale with the original fusion feature map to generate the final Gaussian pyramid attention feature map; S24: input the above attention feature map into the spatial domain attention module, extract the spatial domain key region attention weight through depth separable convolution and standard convolution operation, and finally weightedly fuse the fusion feature map to obtain a multi-scale spatial attention feature map containing key facial regions.

9. The method of claim 8, wherein the method of detecting depression by fusing multi-modal attention mechanism is characterized by, The subsequent process of speech feature processing in S3 is as follows: S31: pre-process the collected speech signal, including noise reduction, endpoint detection and frame operation, and extract the mel spectrum as the speech input feature; S32: input the mel spectrum into the SincNet network to extract the speech feature map containing intonation, speech rate, time domain and frequency domain; S33: input the extracted speech feature map into the time-frequency attention module, connect two attentions using the residual idea to enhance the time and frequency dimension information of the speech, and generate a time-frequency weighting map through the Sigmoid function; S34: element-wise weight the time-frequency weighting map and the original speech feature map to obtain the final speech attention feature map for subsequent fusion.

10. The method of claim 9, wherein the method of detecting depression by fusing multi-modal attention mechanism is characterized by, The process of the multi-modal adaptive attention fusion module in S4 is as follows: S41: input the facial feature map fused by the Gaussian pyramid and the spatial domain attention, and the speech feature map enhanced by the time-frequency attention into the self-attention mechanism channel respectively to capture the long-term dependence relationship within the modal; S42: use the Query-Key-Value structure to build a cross-modal attention path to realize dynamic matching and complementary representation between speech and visual information; S43: use the adaptive control gate mechanism to give dynamic weight coefficients to the two modalities, and fuse to generate a unified multi-modal emotion feature map; S44: integrate the cross-modal attention, with Query coming from the facial feature self-attention, Key and Value coming from the speech feature self-attention; S45: finally, add the visual self-attention and the speech attention guided by the visual attention to obtain the final cross-modal feature.

Images