Depression detection method and device based on multi-modal feature fusion

Abstract

The invention discloses a depression detection method and device based on multi-modal feature fusion. The method comprises the following steps: S1, acquiring a multi-modal depression data set; s2, converting the audio features and the video features into a space of the same dimension for alignment; s3, performing feature enhancement on the converted audio features and video features, and adding the aligned audio features and video features with the feature enhanced features to form residual connection; s4, splicing the enhanced audio features and video features according to the sample length, and carrying out feature fusion on the spliced audio features and video features; s5, the fused features are multiplied by masks for average pooling, a linear layer is used as a classification head, binary cross entropy loss with logic output is used as a loss function, and a predicted value is obtained; and S6, visualizing an evaluation result by adopting a confusion matrix thermodynamic diagram. According to the invention, high-precision depression detection can be realized while the cost is reduced, and visual display of a detection result is supported.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal fusion and depression detection, and more specifically, to a depression detection method and device based on multimodal feature fusion. Background Art

[0002] Depression is a common mental illness that negatively affects patients' thinking, feelings, and behaviors. It does not directly lead to death, but many patients with depression choose to self-harm or even commit suicide because they cannot overcome psychological barriers. Depressed patients usually show behaviors such as social withdrawal, reduced communication with others, and difficulty concentrating, which also poses a challenge to doctors' diagnosis. About 322 million people suffer from depression worldwide, accounting for 4.4% of the total population, but the treatment rate is still low. The high time and cost of treatment, as well as patients concealing their true symptoms due to social prejudice, often lead to misdiagnosis and delayed treatment. Therefore, efficient and accurate automatic depression detection methods are of great significance for early screening of depression and preventing the deterioration of the disease.

[0003] Existing methods have their own advantages, but they still have some shortcomings in efficient long-term time series modeling: Convolutional neural networks (CNNs) are good at extracting local features, but are weak in capturing global context. Recurrent neural networks (RNNs) perform well in modeling sequence data, but have low computational efficiency and poor scalability. The self-attention (SA) mechanism has global modeling capabilities, but has high computational complexity. In addition, existing methods do not consider directional information, which may lead to insufficient perception of features. Summary of the invention

[0004] The purpose of the present invention is to provide a depression detection method and device based on multimodal feature fusion to overcome the defects of the prior art.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A depression detection method based on multimodal feature fusion, comprising:

[0007] S1. Obtain a multimodal depression dataset, which includes audio features, video features, and corresponding label information;

[0008] S2, transforming the audio features and video features in the data set into a space of the same dimension for alignment;

[0009] S3, performing feature enhancement on the transformed audio features and video features, and adding the aligned audio features and video features to the enhanced features to form a residual connection to obtain enhanced features;

[0010] S4, splicing the enhanced audio features and video features according to the sample length, and sending the spliced ​​audio features and video features to a multi-head attention module including a residual connection for feature fusion;

[0011] S5. Multiply the fused features by the mask for average pooling, use the linear layer as the classification head and the binary cross entropy loss with logical output as the loss function to get the predicted value.

[0012] S6. Use confusion matrix heat map to visualize the evaluation results.

[0013] Furthermore, the speech and audio features in step S1 are extracted using the opensmile tool, and the video features are extracted using the dlib tool. In each training batch, samples of different lengths are padded to be consistent with the longest sample and a mask is generated.

[0014] Furthermore, in step S2, one-dimensional convolution is used to transform the audio features and video features in the data set into a space of the same dimension for alignment, and the formula is:

[0015] X′ m =Conv1D(X m )

[0016] In the formula, each sample before alignment can be expressed as: m={a,v} represents the mode of the data, a represents audio, v represents video, T represents the sequence length, D m Represents the feature dimension.

[0017] Furthermore, in step S3, a bidirectional Mamba2 module is used to enhance the aligned features, and then a residual connection is performed, and the formula is:

[0018] X Em =Mamba2(X′ m )+Flip(Mamba2(Flip(X′ m )))+x′ m

[0019] In the formula, Flip means flipping, and the state space dual SSD in Mamba2 is expressed as:

[0020] Y=SSM(A,B,C)(X)=MX

[0021]

[0022] Where SSM represents the state space model, and the input sequence X∈R T×D , vector A∈R T , the matrix B∈R T×N ,C∈RT×N ,M∈R T×T , T represents the sequence length, N represents the state dimension, and D represents the head dimension.

[0023] Furthermore, the step S4 is specifically as follows:

[0024] Concatenate the enhanced features X by feature dimension c =Concate(X Ea ,X Ev ), where X Ea represents the enhanced audio features, X Ev Represents the enhanced video features;

[0025] The concatenated features are fed into a multi-head attention module with residual connections:

[0026] X mh =MHA(X c )+X c

[0027] Where MHA represents the standard multi-head attention.

[0028] Furthermore, the formula for the predicted value in step S5 is:

[0029]

[0030] Use binary cross entropy loss with logistic output as the loss function:

[0031]

[0032] In the formula, y is the true label, To predict the label, the activation function

[0033] The present invention also provides a device for implementing the above-mentioned depression detection method based on multimodal feature fusion, comprising:

[0034] An acquisition module is used to acquire a multimodal depression dataset, which includes audio features, video features, and corresponding label information;

[0035] An alignment module, used for transforming the audio features and video features in the data set into a space of the same dimension for alignment;

[0036] A feature enhancement module is used to enhance the transformed audio features and video features, and add the aligned audio features and video features to the enhanced features to form a residual connection to obtain enhanced features;

[0037] A splicing module, used for splicing the enhanced audio features and video features according to the sample length, and sending the spliced ​​audio features and video features to a multi-head attention module including a residual connection for feature fusion;

[0038] The prediction module is used to multiply the fused features by the mask for average pooling, use the linear layer as the classification head and the binary cross entropy loss with logistic output as the loss function to get the predicted value;

[0039] Evaluation module, which uses confusion matrix heat map to visualize evaluation results;

[0040] The acquisition module, alignment module, feature enhancement module, splicing module, prediction module and evaluation module are connected in sequence.

[0041] Compared with the prior art, the advantages of the present invention are: the present invention applies the Mamba2 model to the field of depression detection, solves the shortcomings of models such as convolutional neural networks, recurrent neural networks and self-attention in efficient modeling, and further uses the bidirectional Mamba2 model to capture directional information, thereby improving the model's modeling ability for global contextual information. In order to better integrate information from multiple modalities, the present invention uses standard multi-head attention to fuse audio and video features, so the present invention achieves the most advanced effect on depression datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a flow chart of the depression detection method based on multimodal feature fusion of the present invention.

[0044] Figure 2 It is an overall framework diagram of the depression detection method based on multimodal feature fusion of the present invention.

[0045] Figure 3 This is a diagram of the Mamba2 model in the depression detection method based on multimodal feature fusion of the present invention.

[0046] Figure 4 This is a visualization diagram of the prediction results of the depression detection method based on multimodal feature fusion of the present invention.

[0047] Figure 5 It is a schematic diagram of a depression detection device based on multimodal feature fusion of the present invention. DETAILED DESCRIPTION

[0048] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0049] See also Figure 1 As shown, this embodiment discloses a depression detection method based on multimodal feature fusion, including:

[0050] Step S1: Obtain a multimodal depression dataset, which includes audio features, video features, and corresponding label information.

[0051] In order to extract audio features, this embodiment uses the OpenSmile tool to extract the extended Geneva simplified acoustic parameter set (eGeMAPS), and the specific steps are as follows: first, each audio sample is divided into t segments by second; then, 25 low-level acoustic descriptors (LLDs) are extracted from each audio segment, including loudness, Mel frequency cepstral coefficients (MFCCs), spectral flux, etc., and the frame size and frame step size are set to 0.06 seconds and 0.01 seconds respectively; finally, the 25 feature values ​​in each audio segment are averaged, and the feature values ​​of consecutive t segments are connected to obtain an audio feature vector. In visual feature extraction, this embodiment uses the dlib tool. 68 facial key points (i.e., x and y coordinates) are extracted from each frame of the video sample (1 frame is processed per second), and a 136-dimensional vector is generated by connecting the x and y coordinates of each key point. If a frame does not detect a face, it is filled with a full zero vector. Finally, the continuous features in the sample are connected to obtain a video feature vector. Since the time span of each sample is inconsistent, there are differences in sample length. In each training batch, samples of different lengths are padded to be consistent with the longest sample, and the corresponding masks are generated at the same time. A mask value of 0 indicates that the data at that position is a padded value, while a mask value of 1 indicates that the data at that position is valid data.

[0052] Step S2: Since the feature dimensions of the extracted audio and video data are inconsistent, this embodiment uses one-dimensional convolution to transform the audio features and video features in the data set into a space of the same dimension for alignment. The formula is:

[0053] X′ m =Conv1D(X m )

[0054] In the formula, each sample before alignment can be expressed as: m={a,v} represents the mode of the data, a represents audio, v represents video, T represents the sequence length, D m Represents the feature dimension.

[0055] Step S3: Enhance the transformed audio features and video features, and add the aligned audio features and video features to the enhanced features to form a residual connection to obtain enhanced features.

[0056] This embodiment uses a bidirectional Mamba2 module to enhance the aligned features, and then performs residual connection. The bidirectional processing mechanism enhances its feature perception ability by merging the forward and backward information flows, and fully mines the directional information in the data. The formula for enhancing the aligned features by the bidirectional Mamba2 module is:

[0057] X Em =Mamba2(X′ m )+Flip(Mamba2(Flip(X′ m )))+X′ m

[0058] In the formula, Flip means flip, Mamba2 (such as Figure 3 The state space dual SSD in (shown) is expressed as:

[0059] Y=SSM(A,B,C)(X)=MX

[0060]

[0061] Where SSM represents the state space model, and the input sequence X∈R T×D , vector A∈R T , the matrix B∈R T×N ,C∈R T×N ,M∈R T×T , T represents the sequence length, N represents the state dimension, and D represents the head dimension.

[0062] Step S4: splice the enhanced audio features and video features according to the sample length, and send the spliced ​​audio features and video features to a multi-head attention module containing residual connections for feature fusion.

[0063] In this embodiment, the enhanced features X are connected according to the feature dimension. c =Concate(X Ea ,X Ev ), where X Ea represents the enhanced audio features, X Ev Represents the enhanced video features.

[0064] The concatenated features are fed into a multi-head attention module with residual connections:

[0065] X mh =MHA(X c )+Xc

[0066] Where MHA represents the standard multi-head attention.

[0067] Step S5: Multiply the fused features by the mask (assign the value filled in step S2 to 0) for average pooling, use the linear layer as the classification head and the binary cross entropy loss with logical output as the loss function to get the predicted value, the formula is:

[0068]

[0069] Use binary cross entropy loss with logistic output as the loss function:

[0070]

[0071] In the formula, y is the true label, To predict the label, the activation function

[0072] Step S6: Use confusion matrix heat map to visualize the evaluation results.

[0073] Specifically, in order to verify the effectiveness of the present invention, Figure 4 The visualization results of the network model constructed by the present invention on the data set are given. The true positive rate (TPR) is 86.99%, the false negative rate (FNR) is 13.01%, the false positive rate (FPR) is 37.08%, and the true negative rate (TNR) is 62.92%.

[0074] See also Figure 5 As shown, the present invention also provides a device for implementing the above-mentioned depression detection method based on multimodal feature fusion, including: an acquisition module 1, used to acquire a multimodal depression data set, the data set including audio features, video features and corresponding label information; an alignment module 2, used to transform the audio features and video features in the data set to a space of the same dimension for alignment; a feature enhancement module 3, used to enhance the transformed audio features and video features, and add the aligned audio features and video features to the enhanced features to form a residual connection to obtain enhanced features; a splicing module 4, used to splice the enhanced audio features and video features according to the sample length, and send the spliced ​​audio features and video features to a multi-head attention module containing a residual connection for feature fusion; a prediction module 5, used to multiply the fused features by a mask for average pooling, use a linear layer as a classification head, and a binary cross entropy loss with a logical output as a loss function to obtain a predicted value; an evaluation module 6, using a confusion matrix heat map to visualize the evaluation results; the acquisition module 1, the alignment module 2, the feature enhancement module 4, the splicing module 5, the prediction module 6 and the evaluation module 7 are connected in sequence.

[0075] The present invention applies the Mamba2 model to the field of depression detection, which solves the shortcomings of models such as convolutional neural networks, recurrent neural networks and self-attention in efficient modeling, and further uses the bidirectional Mamba2 model to capture directional information, thereby improving the model's modeling ability for global contextual information. In order to better integrate information from multiple modalities, the present invention uses standard multi-head attention to fuse audio and video features. Therefore, the present invention achieves the most advanced effect on the depression dataset.

[0076] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, the patent owner may make various variations or modifications within the scope of the appended claims. As long as they do not exceed the protection scope described in the claims of the present invention, they should be within the protection scope of the present invention.

Claims

1. A depression detection method based on multimodal feature fusion, characterized in that: include: S1. Obtain a multimodal depression dataset, which includes audio features, video features, and corresponding label information; S2, transforming the audio features and video features in the data set into a space of the same dimension for alignment; S3, performing feature enhancement on the transformed audio features and video features, and adding the aligned audio features and video features to the enhanced features to form a residual connection to obtain enhanced features; S4, splicing the enhanced audio features and video features according to the sample length, and sending the spliced ​​audio features and video features to a multi-head attention module including a residual connection for feature fusion; S5. Multiply the fused features by the mask for average pooling, use the linear layer as the classification head and the binary cross entropy loss with logical output as the loss function to get the predicted value. S6. Use confusion matrix heat map to visualize the evaluation results.

2. The depression detection method based on multimodal feature fusion according to claim 1, characterized in that: The speech and audio features in step S1 are extracted using the opensmile tool, and the video features are extracted using the dlib tool. In each training batch, samples of different lengths are padded to be consistent with the longest sample and a mask is generated.

3. The depression detection method based on multimodal feature fusion according to claim 1, characterized in that: In step S2, one-dimensional convolution is used to transform the audio features and video features in the data set into a space of the same dimension for alignment, and the formula is: X′ m =Conv1D(X m ) In the formula, each sample before alignment can be expressed as: m={a,v} represents the mode of the data, a represents audio, v represents video, T represents the sequence length, D m Represents the feature dimension.

4. The depression detection method based on multimodal feature fusion according to claim 1, characterized in that: In step S3, a bidirectional Mamba2 module is used to enhance the aligned features, and then a residual connection is performed, and the formula is: X Em =Mamba2(X′ m )+Flip(Mamba2(Flip(X′ m )))+X′ m In the formula, Flip means flipping, and the state space dual SSD in Mamba2 is expressed as: Y=SSM(A,B,C)(X)=MX Where SSM represents the state space model, and the input sequence X∈R T×D , vector A∈R T , the matrix B∈R T×N ,C∈R T×N ,M∈R T ×T , T represents the sequence length, N represents the state dimension, and D represents the head dimension.

5. The depression detection method based on multimodal feature fusion according to claim 1, characterized in that: The step S4 is specifically as follows: Concatenate the enhanced features X by feature dimension c =Concate(X Ea ,X Ev ), where X Ea represents the enhanced audio features, X Ev Represents the enhanced video features; The concatenated features are fed into a multi-head attention module with residual connections: X mh =MHA(X c )+X c Where MHA represents the standard multi-head attention.

6. The depression detection method based on multimodal feature fusion according to claim 1, characterized in that: The formula for the predicted value in step S5 is: Use binary cross entropy loss with logistic output as the loss function: In the formula, y is the true label, To predict the label, the activation function 7. A device for implementing the depression detection method based on multimodal feature fusion according to any one of claims 1 to 6, characterized in that: include: An acquisition module is used to acquire a multimodal depression dataset, which includes audio features, video features, and corresponding label information; An alignment module, used for transforming the audio features and video features in the data set into a space of the same dimension for alignment; A feature enhancement module is used to enhance the transformed audio features and video features, and add the aligned audio features and video features to the enhanced features to form a residual connection to obtain enhanced features; A splicing module, used for splicing the enhanced audio features and video features according to the sample length, and sending the spliced ​​audio features and video features to a multi-head attention module including a residual connection for feature fusion; The prediction module is used to multiply the fused features by the mask for average pooling, use the linear layer as the classification head and the binary cross entropy loss with logistic output as the loss function to get the predicted value; Evaluation module, which uses confusion matrix heat map to visualize evaluation results; The acquisition module, alignment module, feature enhancement module, splicing module, prediction module and evaluation module are connected in sequence.

Images