Multi-modal depression detection method and device based on state space model

Abstract

The invention discloses a depression multi-mode detection method and device based on a state space model. The method comprises the following steps: reading a face video sequence and an audio sequence of a detected person as input data; extracting spatial features to form a video spatial feature vector and an audio spatial feature vector; a state space model based on a Mama framework is adopted to learn the time sequence features, and a video time sequence feature vector and an audio time sequence feature vector are obtained; a multi-modal fusion module is used for carrying out conjoint analysis on the spatial features and the time sequence features, multi-modal spatial-temporal feature vectors are generated through splicing, and then dynamic weighted feature fusion is achieved through a self-attention mechanism to obtain fusion features; and the output module generates a final depression detection result through multi-layer full-connection network processing based on the fusion features. According to the method, the problems of insufficient diagnosis accuracy and robustness caused by information loss, insufficient multi-mode integration and poor real scene adaptability when long sequence data is processed by the existing depression detection method are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing and analysis, and in particular to a multimodal depression detection method and device based on a state-space model. Background Art

[0002] Depression is a mental disorder that severely impacts patients' physical and mental health and quality of life. Early detection is crucial for effective treatment. Traditionally, the diagnosis of depression relies on face-to-face interactions between mental health professionals and patients to assess their emotional state, including observing facial expressions, body language, and listening to verbal expressions. However, this approach relies heavily on the physician's experience and personal judgment, and can easily overlook subtle emotional changes due to subjectivity, leading to misdiagnosis. With the development of artificial intelligence (AI) technology, AI-assisted automated depression detection methods are gaining popularity. These methods can extract and analyze a variety of information from patient videos and voice samples, making depression diagnosis more objective and systematic. For example, detection systems can identify depression-related behaviors from facial expression videos, such as the frequency and intensity of smiles, as well as vocal characteristics such as intonation, speed, and pauses. Combined with a series of scales and algorithms, these systems can then perform efficient and automated depression detection.

[0003] Although existing AI automated detection methods are highly efficient, they still present challenges in certain aspects. First, when processing long time series data, there are often problems with information forgetting and loss, which makes it difficult for the model to accurately interpret long video and long audio data, affecting the detection performance of the model. In addition, the diagnosis of depression usually requires the integration of data from multiple sensory modalities, but current models have difficulty in effectively integrating data from different modalities. This isolation effect between modalities limits the comprehensive understanding of the patient's condition, thereby affecting the depth and accuracy of the diagnosis. Traditional methods usually use simple splicing or weighted averaging in the multimodal fusion stage, lacking the ability to dynamically recognize key features, resulting in important information (such as micro-expressions in videos and sudden changes in tone in audio) being ignored. When processing high-resolution videos or long audio, existing models are limited by computational complexity and find it difficult to simultaneously take into account local details and global contextual relationships, resulting in incomplete feature extraction. Summary of the Invention

[0004] To this end, the present invention provides a multimodal depression detection method and device based on a state-space model to solve the problems of information loss, insufficient multimodal integration, and poor adaptability to real scenarios when processing long sequence data in existing AI depression detection methods, resulting in insufficient diagnostic accuracy and robustness.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multimodal detection method for depression based on a state-space model, comprising the following steps:

[0006] Read the subject's facial video sequence and audio sequence as input data through the input module;

[0007] Using a spatial feature extraction module to extract spatial features from the facial video sequence and the audio sequence respectively to form a video spatial feature vector and an audio spatial feature vector;

[0008] With the help of a temporal feature learning module, a state space model based on the Mamba architecture is used to perform temporal feature learning on the facial video sequence and the audio sequence respectively to obtain a video temporal feature vector and an audio temporal feature vector;

[0009] A multimodal fusion module is used to jointly analyze the spatial features and temporal features from the facial video sequence and the audio sequence, first splicing to generate a multimodal spatiotemporal feature vector, and then dynamically weighted feature fusion is achieved through a self-attention mechanism to obtain a fused feature;

[0010] The output module generates the final depression detection result based on the fusion features after processing through a multi-layer fully connected network.

[0011] As a preferred solution of the multimodal depression detection method based on the state space model, the spatial feature extraction module extracts spatial features, specifically including:

[0012] The facial video sequence is passed through a convolutional neural network to obtain a video feature map, and then a global pooling based on an attention mechanism is performed on the video feature map in the time step dimension to obtain the video spatial feature vector;

[0013] The audio sequence is passed through a convolutional neural network to obtain an audio feature map, and global pooling based on an attention mechanism is performed on the audio feature map in the time step dimension to obtain the audio spatial feature vector.

[0014] As a preferred solution of the multimodal depression detection method based on the state-space model, when the temporal feature learning module performs temporal feature learning on the facial video sequence, the specific process is as follows:

[0015] Split the video into T frames and divide each frame into N s ×N s non-overlapping blocks;

[0016] Initialize the state vector and position encoding vector for each block of each frame;

[0017] Perform forward and backward scans separately to capture temporal dependencies;

[0018] The forward and backward states are concatenated to form a final feature, and then global average pooling is performed on all block features to obtain the video temporal feature vector of the facial video sequence.

[0019] As a preferred solution of the multimodal depression detection method based on the state-space model, when the temporal feature learning module performs temporal feature learning on the audio sequence, the specific steps include:

[0020] Divide the audio signal into multiple frames and apply a Hamming window to the frames at each time step to ensure signal continuity;

[0021] Mel-frequency cepstral coefficients are used to extract acoustic spectrum features for each frame;

[0022] Calculate the hidden state of the time step using a time-varying parameter matrix that is dynamically adjusted based on the input data and generate the output feature vector for the current time step;

[0023] The outputs of all time steps are averaged to obtain the final representation vector of the audio modality.

[0024] As a preferred solution of the multimodal depression detection method based on the state-space model, the specific operations of using the multimodal fusion module to jointly analyze the spatial features and temporal features from the facial video sequence and the audio sequence include:

[0025] Concatenating the video spatial feature vector with the video temporal feature vector to generate a spatiotemporal embedding of the facial video sequence, and concatenating the audio spatial feature vector with the audio temporal feature vector to generate a spatiotemporal embedding of the audio sequence;

[0026] splicing the spatiotemporal embedding of the facial video sequence and the spatiotemporal embedding of the audio sequence to generate a multimodal spatiotemporal feature vector;

[0027] Calculate the query-key-value mapping matrix of the multimodal spatiotemporal feature vector and perform dynamic weighted feature fusion.

[0028] As a preferred solution of the multimodal depression detection method based on the state-space model, the specific steps of generating the final depression detection result after processing the fusion features through a multi-layer fully connected network via the output module are as follows:

[0029] The multimodal spatiotemporal feature vector is mapped to a high-dimensional space through a fully connected layer and processed by a nonlinear activation function and a dropout function;

[0030] The high-dimensional vector is then mapped back to the low-dimensional space through the fully connected layer, and finally the Softmax activation function is used to obtain the probability distribution of multiple categories, and the category with the highest probability is taken as the final detection result.

[0031] The present invention also provides a multimodal depression detection device based on a state-space model, comprising:

[0032] An input module, used for reading the subject's facial video sequence and audio sequence as input data;

[0033] A spatial feature extraction module, configured to extract spatial features from the facial video sequence and the audio sequence, respectively, to form a video spatial feature vector and an audio spatial feature vector;

[0034] A temporal feature learning module, configured to perform temporal feature learning on the facial video sequence and the audio sequence respectively using a state space model based on the Mamba architecture, to obtain a video temporal feature vector and an audio temporal feature vector;

[0035] A multimodal fusion module is used to jointly analyze the spatial features and temporal features from the facial video sequence and the audio sequence, first splicing to generate a multimodal spatiotemporal feature vector, and then performing dynamic weighted feature fusion through a self-attention mechanism to obtain a fused feature;

[0036] The output module is used to generate a final depression detection result based on the fusion features after processing through a multi-layer fully connected network.

[0037] As a preferred solution of the multimodal depression detection device based on the state space model, the spatial feature extraction module includes:

[0038] A video spatial feature vector generation submodule is used to obtain a video feature map by passing a convolutional neural network through the facial video sequence, and then perform global pooling based on the attention mechanism on the video feature map in the time step dimension to obtain the video spatial feature vector;

[0039] The audio spatial feature vector generation submodule is used to obtain an audio feature map by passing the audio sequence through a convolutional neural network, and perform global pooling based on the attention mechanism on the audio feature map in the time step dimension to obtain the audio spatial feature vector.

[0040] As a preferred solution of the multimodal depression detection device based on the state space model, the temporal feature learning module includes:

[0041] The video frame and block submodule is used to divide the video into T frame sequences and divide each frame into N s ×N s non-overlapping blocks;

[0042] The state and position code initialization submodule is used to initialize the state vector and position code vector for each block of each frame;

[0043] A bidirectional time dependency capture submodule, configured to perform forward scanning and backward scanning to capture time dependency;

[0044] A feature fusion pooling submodule concatenates the forward and backward states to form a final feature, and then performs global average pooling on all block features to obtain the video temporal feature vector of the facial video sequence;

[0045] The audio framing and windowing submodule is used to divide the audio signal into multiple frames and apply a Hamming window to the frames at each time step to ensure signal continuity;

[0046] Acoustic spectrum feature extraction submodule, used to extract acoustic spectrum features for each frame using Mel-frequency cepstral coefficients;

[0047] The dynamic state update submodule is used to calculate the hidden state of the time step using a time-varying parameter matrix that is dynamically adjusted based on the input data and generate the output feature vector of the current time step;

[0048] The time series feature averaging submodule is used to average the outputs of all time steps to obtain the final representation vector of the audio modality.

[0049] As a preferred solution of the multimodal detection device for depression based on the state-space model, the multimodal fusion module includes:

[0050] a video spatiotemporal embedding generation submodule, configured to concatenate the video spatial feature vector and the video temporal feature vector to generate a spatiotemporal embedding of the facial video sequence;

[0051] An audio spatiotemporal embedding generation submodule, which concatenates the audio spatial feature vector and the audio temporal feature vector to generate a spatiotemporal embedding of the audio sequence;

[0052] a cross-modal feature splicing submodule, configured to splice the spatiotemporal embedding of the facial video sequence and the spatiotemporal embedding of the audio sequence to generate a multimodal spatiotemporal feature vector;

[0053] Dynamic weighted fusion submodule, used to calculate the query-key-value mapping matrix of multimodal spatiotemporal feature vectors and perform dynamic weighted feature fusion;

[0054] The output module includes:

[0055] The high-dimensional feature extraction submodule is used to map the multimodal spatiotemporal feature vectors to a high-dimensional space through a fully connected layer and process them through a nonlinear activation function and a dropout function;

[0056] The classification result generation submodule is used to map the high-dimensional vector back to the low-dimensional space through the fully connected layer, and finally obtain the probability distribution of multiple categories through the Softmax activation function, and take the category with the highest probability as the final detection result.

[0057] The present invention has the following advantages:

[0058] First, the input module reads multimodal facial video and audio data, avoiding the limitations of single-modality diagnosis. Compared with traditional methods that rely on doctors' subjective judgment (such as observing facial expressions and listening to verbal expressions), this technology analyzes multi-dimensional features such as the frequency of micro-expressions in videos and changes in audio intonation, shifting depression detection from subjective experience to objective data-driven, significantly reducing the misdiagnosis rate.

[0059] Second, the hierarchical design of the spatial feature extraction module and the temporal feature learning module solves the information loss problem when traditional models process long sequence data;

[0060] Third, the multimodal fusion module achieves dynamic weighted fusion through a self-attention mechanism, avoiding feature redundancy caused by traditional simple splicing. This mechanism quantifies the correlation between video and audio features, focusing on key time points, enabling the model to provide a more comprehensive analysis of the patient's mental state. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0062] The structures, proportions, sizes, etc. illustrated in this specification are intended solely to complement the contents disclosed herein and to facilitate understanding and reading by persons skilled in the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall remain within the scope of the technical contents disclosed herein.

[0063] Figure 1 Schematic diagram of the process of a multimodal depression detection method based on a state-space model provided in an embodiment of the present invention;

[0064] Figure 2 Schematic diagram of the architecture of a multimodal depression detection device based on a state-space model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0066] Example 1

[0067] See also Figure 1 Embodiment 1 of the present invention provides a multimodal depression detection method based on a state-space model, comprising the following steps:

[0068] Step 1: Read the subject's facial video sequence and audio sequence as input data through the input module;

[0069] Step 2: Using a spatial feature extraction module, extract spatial features from the facial video sequence and the audio sequence respectively to form a video spatial feature vector and an audio spatial feature vector;

[0070] Step 3: Using a temporal feature learning module, a state space model based on the Mamba architecture is used to perform temporal feature learning on the facial video sequence and the audio sequence, respectively, to obtain a video temporal feature vector and an audio temporal feature vector;

[0071] Step 4: Use the multimodal fusion module to jointly analyze the spatial features and temporal features from the facial video sequence and the audio sequence, first splicing to generate a multimodal spatiotemporal feature vector, and then using the self-attention mechanism to achieve dynamic weighted feature fusion to obtain a fused feature;

[0072] Step 5: The output module generates the final depression detection result based on the fusion features and processed through a multi-layer fully connected network.

[0073] In this embodiment, step 1 specifically includes:

[0074] Step 101: Read the subject's facial video sequence X v As the first input;

[0075] Step 102: Read the subject's audio sequence X a As the second input;

[0076] In this embodiment, when the spatial feature extraction module extracts the spatial features, it specifically includes:

[0077] The facial video sequence is passed through a convolutional neural network to obtain a video feature map, and then a global pooling based on an attention mechanism is performed on the video feature map in the time step dimension to obtain the video spatial feature vector;

[0078] The audio sequence is passed through a convolutional neural network to obtain an audio feature map, and global pooling based on an attention mechanism is performed on the audio feature map in the time step dimension to obtain the audio spatial feature vector.

[0079] Specifically, in step 2, the spatial feature learning module is responsible for extracting spatial features from the input original video and audio data. When the spatial feature extraction module extracts spatial features, the steps are as follows:

[0080] Step 201: Input facial video sequence X v Through convolutional neural network CNN v , get the video feature map

[0081] F v =CNN v (X v )

[0082] Step 202: Video feature map F v Perform Attention-Guided Global Pooling (AGGP) based on the time step T to obtain a fixed-length video feature vector As the spatial features of the subject's facial video:

[0083] Q v =Linear(F v ),K v =Linear(F v ),V v =Linear(F v )

[0084]

[0085] Where Q v , K v 、V v are the query, key, and value matrices of the spatial features of the video, generated by linear transformation; d is the scaling factor used to adjust the calculation scale of the attention weight;

[0086] Step 203: Input audio sequence X a Through convolutional neural network CNN a , get the audio feature map

[0087] F a =CNN a (X a )

[0088] Step 204: Then, global pooling based on the attention mechanism is performed on the time step T to obtain the speech space feature vector As the spatial characteristics of audio signals.

[0089] Q a =Linear(F a ),K a =Linear(F a ),V a =Linear(F a )

[0090]

[0091] Where Q a , K a 、V a are the query, key, and value matrices of the spatial features of the audio signal, generated by linear transformation; d is the scaling factor used to adjust the calculation scale of the attention weight.

[0092] The spatial representation of video and audio data is obtained by compressing spatial features into fixed-length feature vectors.

[0093] In this embodiment, in step 3, when the temporal feature learning module performs temporal feature learning on the facial video sequence, the specific process is as follows:

[0094] Split the video into T frames and divide each frame into N s ×N s non-overlapping blocks;

[0095] Initialize the state vector and position encoding vector for each block of each frame;

[0096] Perform forward and backward scans separately to capture temporal dependencies;

[0097] The forward and backward states are concatenated to form a final feature, and then global average pooling is performed on all block features to obtain the video temporal feature vector of the facial video sequence.

[0098] Specifically, the core of the temporal feature learning module in step 3 is the state space model (SSM) based on the Mamba architecture. VisionMamba is used to capture the temporal dynamic features of the video, while Mamba2 is used to capture the temporal dynamic features of the audio data. The specific steps are as follows:

[0099] Step 3011: Video Xv Divided into T frame sequence, X v =[x1 v ,x2 v ,…,x T v ];

[0100] Step 3012: Divide each frame into N s ×N s non-overlapping blocks, the total number of blocks is N = N s ×N s , the i-th block of the frame at time t is represented as x t,i ;

[0101] Step 3013: Initialize the state vector for each block of each frame and the position encoding vector

[0102] Step 3014: Perform forward scan to capture time dependency:

[0103] Step 3015: Perform a backward scan to capture temporal reverse dependencies:

[0104] Step 3016: Concatenate the forward and backward states into the final feature:

[0105] Step 3017: Perform global average pooling on all block features to obtain the temporal feature vector of the video modality

[0106] In this embodiment, in step 3, when the temporal feature learning module performs temporal feature learning on the audio sequence, the specific steps include:

[0107] Divide the audio signal into multiple frames and apply a Hamming window to the frames at each time step to ensure signal continuity;

[0108] Mel-frequency cepstral coefficients are used to extract acoustic spectrum features for each frame;

[0109] Calculate the hidden state of the time step using a time-varying parameter matrix that is dynamically adjusted based on the input data and generate the output feature vector for the current time step;

[0110] The outputs of all time steps are averaged to obtain the final representation vector of the audio modality.

[0111] Specifically, in step 3, Mamba2 is used to capture the temporal dynamic characteristics of the audio data. The steps are as follows:

[0112] Step 3021: Convert the audio signal X a Divided into T frame, X a =[x1,x2,…,x T ];

[0113] Step 3022: Apply a Hamming window to each time step frame to ensure signal continuity, x t =x t ω;

[0114] Step 3023: For each frame, use Mel-Frequency Cepstral Coefficients (MFCC) to extract acoustic spectrum features, x t =MFCC(x t );

[0115] Step 3024: Use the time-varying parameter matrix A that is dynamically adjusted based on the input data t 、B t , to calculate the hidden state at time step t

[0116] h t =A t ·h t-1 +B t ·x t

[0117] Step 3025: According to the hidden state h t Generate the output feature vector for the current time step

[0118] y t =Ch t +Dx t

[0119] Step 3026: Take the average of the outputs of all time steps to obtain the final representation vector e of the audio mode a :

[0120]

[0121] Mamba2 models the dependencies of the time dimension in the audio modality and outputs the temporal feature vector of the audio modality.

[0122] In this embodiment, in step 4, the specific operations of using the multimodal fusion module to jointly analyze the spatial features and temporal features from the facial video sequence and the audio sequence include:

[0123] Concatenating the video spatial feature vector with the video temporal feature vector to generate a spatiotemporal embedding of the facial video sequence, and concatenating the audio spatial feature vector with the audio temporal feature vector to generate a spatiotemporal embedding of the audio sequence;

[0124] splicing the spatiotemporal embedding of the facial video sequence and the spatiotemporal embedding of the audio sequence to generate a multimodal spatiotemporal feature vector;

[0125] Calculate the query-key-value mapping matrix of the multimodal spatiotemporal feature vector and perform dynamic weighted feature fusion.

[0126] Specifically, in step 4, the multimodal fusion module jointly analyzes the spatial features and temporal features from the video and audio modalities as follows:

[0127] Step 401: Combine the spatial features and temporal features of the video to generate the spatiotemporal embedding m of the video modality v =[S v :e v ];

[0128] Step 402: Combine the spatial and temporal features of the audio to generate a spatiotemporal embedding m of the speech modality. a =[S a :e a ];

[0129] Step 403: embed the spatiotemporal features of the video and audio modalities and further concatenate them to generate a multimodal spatiotemporal feature vector m = [m v :m a ].

[0130] Step 404: Use the self-attention mechanism to perform dynamic weighted feature fusion. First, calculate the query-key-value mapping matrix of the spatiotemporal feature vector m:

[0131] Q=W q m,K=W k m,V=W v m

[0132] Step 405: Calculate the weighted fusion feature m':

[0133]

[0134] Among them, W q 、W k 、W v is the weight matrix used to generate query, key, and value vectors; d k is the dimension of the key vector, used to scale the attention scores.

[0135] In this embodiment, in step 5, the specific steps of generating the final depression detection result after the output module processes the fusion features through a multi-layer fully connected network are as follows:

[0136] The multimodal spatiotemporal feature vector is mapped to a high-dimensional space through a fully connected layer and processed by a nonlinear activation function and a dropout function;

[0137] The high-dimensional vector is then mapped back to the low-dimensional space through the fully connected layer, and finally the Softmax activation function is used to obtain the probability distribution of multiple categories, and the category with the highest probability is taken as the final detection result.

[0138] Specifically, in step 5, the output module generates the final detection result based on the fusion feature m' through a multi-layer fully connected network:

[0139] Step 501: Use the fully connected layer FC1 to map the multimodal spatiotemporal feature vector m' to the high-dimensional space z1 to extract deeper and more abstract features:

[0140] z1=W1m'+b1=FC1(m')

[0141] Step 502: Input z1 into the nonlinear activation function ReLU to obtain the output r1, breaking the linear relationship and extracting a more complex nonlinear feature combination:

[0142] r1=max(0,z1)=ReLU(z1)

[0143] Step 503: Input r1 into the dropout function to obtain output d1, reducing the complexity of the model and increasing the generalization ability of the model:

[0144] d1=Dropout(r1)

[0145] Step 504: Use the fully connected layer FC2 to map the high-dimensional vector back to the low-dimensional space o to facilitate further classification processing:

[0146] o=W2d1+b2=FC2(d1)

[0147] Step 505: Finally, use the Softmax activation function to map the prediction vector into the probability distribution of C categories. Output the probability value of each category, and the category with the highest probability is the final predicted category.

[0148]

[0149] Among them, W2 and b2 are the weight matrix and bias term of FC2.

[0150] In one possible embodiment, the present invention takes multimodal data as input, including facial videos (extracting facial expressions and key point changes) and speech signals (analyzing features such as voice intonation, speed, and pauses). First, local features are extracted through a spatial feature learning module, and a temporal feature learning module captures the temporal dynamic characteristics of the data; then, a multimodal fusion module is used to comprehensively analyze the spatial and temporal information of the multimodal data to improve the diagnostic accuracy and robustness of the model. In the data preprocessing stage, the audio data sampling rate is 16kHz, and the video data is processed at a frame rate of 30fps. The Adam optimizer is used in the model training stage, with a learning rate of 1e-4, a batch size of 128, and a maximum training cycle of 70 cycles.

[0151] The implementation effects of the present invention are as follows:

[0152] By introducing the bidirectional state space model (B-SSM) and dynamic state update mechanism based on VisionMamba and Mamba2, the modeling capability of long time series data of video and audio has been significantly improved. VisionMamba uses serialized small block processing and spatiotemporal scanning strategies to efficiently capture local details and global contextual relationships between video frames with O(LlogL) complexity, solving the problem of key feature loss in long videos by traditional methods. Mamba2's dynamic state update mechanism adapts to the non-stationary nature of speech signals, improves the robustness of audio features at low computational overhead, and retains more temporal information related to depression.

[0153] The present invention combines convolutional neural networks (CNN) with VisionMamba / Mamba2 to extract spatial and temporal features from facial videos and voice signals, respectively, and implements multimodal dynamic weighted fusion through a self-attention mechanism. Through the self-attention mechanism and dynamic selection mechanism, redundant information in multimodal data is effectively filtered out, significantly improving the detection accuracy and robustness of the model. The experimental results of the proposed method on the D-Vlog dataset compared with a variety of advanced algorithms show excellent performance. Specific data are shown in Table 1 below:

[0154]

[0155]

[0156] Table 1 Comparative experimental results of the present invention and the specified algorithm on the D-Vlog dataset

[0157] The precision of our method reached 75.79%, significantly higher than DepMSTAT's 71.53%. Furthermore, our method achieved an F1 score of 75.66%, surpassing all other methods. Its closest competitor, DepMSTAT, achieved an F1 score of only 73.51%. These results clearly demonstrate that our method has superior performance and robustness in depression detection tasks compared to other methods, enabling more accurate depression detection in complex data environments.

[0158] It can be seen that the present invention adopts a multi-layer feature extraction and fusion strategy, combined with the spatiotemporal scanning strategy of VisionMamba, the dynamic state update and self-attention mechanism of Mamba2, to deeply mine the multidimensional features in video and audio data. The model has a strong ability to distinguish different degrees of depression and is suitable for actual clinical auxiliary diagnosis. The present invention applies VisionMamba and Mamba2 to depression detection tasks for the first time, combines multimodal data processing methods, and expands the application range of structured state space models (SSM) in the medical field. The modular design of the present invention also provides a good foundation for subsequent expansion of other data modalities (such as text, EEG signals).

[0159] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario, where multiple devices cooperate with each other to complete the method. In this distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the multimodal detection method for depression based on the state-space model.

[0160] It should be noted that the above description is limited to some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0161] Example 2

[0162] See also Figure 2 Embodiment 2 of the present invention further provides a multimodal depression detection device based on a state-space model, comprising:

[0163] Input module 001, used to read the subject's facial video sequence and audio sequence as input data;

[0164] A spatial feature extraction module 002 is configured to extract spatial features from the facial video sequence and the audio sequence, respectively, to form a video spatial feature vector and an audio spatial feature vector;

[0165] A temporal feature learning module 003 is configured to perform temporal feature learning on the facial video sequence and the audio sequence respectively using a state space model based on the Mamba architecture to obtain a video temporal feature vector and an audio temporal feature vector;

[0166] The multimodal fusion module 004 is used to jointly analyze the spatial features and temporal features from the facial video sequence and the audio sequence, first splicing to generate a multimodal spatiotemporal feature vector, and then performing dynamic weighted feature fusion through a self-attention mechanism to obtain a fused feature;

[0167] The output module 005 is used to generate the final depression detection result based on the fusion features after processing through a multi-layer fully connected network.

[0168] In this embodiment, the spatial feature extraction module 002 includes:

[0169] The video spatial feature vector generation submodule 021 is used to obtain a video feature map by passing a convolutional neural network through the facial video sequence, and then perform global pooling based on the attention mechanism on the video feature map in the time step dimension to obtain the video spatial feature vector;

[0170] The audio spatial feature vector generation submodule 022 is used to obtain an audio feature map by passing the audio sequence through a convolutional neural network, and perform global pooling based on the attention mechanism on the audio feature map in the time step dimension to obtain the audio spatial feature vector.

[0171] In this embodiment, the time series feature learning module 003 includes:

[0172] The video frame and block submodule 031 is used to divide the video into T frame sequences and divide each frame into N s ×N s non-overlapping blocks;

[0173] A state and position code initialization submodule 032 is used to initialize a state vector and a position code vector for each block of each frame;

[0174] A bidirectional time dependency capturing submodule 033 is configured to perform forward scanning and backward scanning to capture the time dependency relationship;

[0175] The feature fusion pooling submodule 034 concatenates the forward and backward states to form the final feature, and then performs global average pooling on all block features to obtain the video time series feature vector of the facial video sequence;

[0176] The audio framing and windowing submodule 035 is used to divide the audio signal into multiple frames and apply a Hamming window to the frames at each time step to ensure signal continuity;

[0177] Acoustic spectrum feature extraction submodule 036, for extracting acoustic spectrum features for each frame using Mel-frequency cepstral coefficients;

[0178] Dynamic state update submodule 037, for calculating the hidden state of the time step using a time-varying parameter matrix dynamically adjusted based on input data, and generating an output feature vector of the current time step;

[0179] The time series feature averaging submodule 038 is used to average the outputs of all time steps to obtain the final representation vector of the audio mode.

[0180] In this embodiment, the multimodal fusion module 004 includes:

[0181] A video spatiotemporal embedding generation submodule 041 is configured to concatenate the video spatial feature vector and the video temporal feature vector to generate a spatiotemporal embedding of the facial video sequence;

[0182] The audio spatiotemporal embedding generation submodule 042 concatenates the audio spatial feature vector and the audio temporal feature vector to generate a spatiotemporal embedding of the audio sequence;

[0183] A cross-modal feature splicing submodule 043 is used to splice the spatiotemporal embedding of the facial video sequence and the spatiotemporal embedding of the audio sequence to generate a multimodal spatiotemporal feature vector;

[0184] Dynamic weighted fusion submodule 044 is used to calculate the query-key-value mapping matrix of the multimodal spatiotemporal feature vector and perform dynamic weighted feature fusion;

[0185] The output module 005 includes:

[0186] High-dimensional feature extraction submodule 051 is used to map the multimodal spatiotemporal feature vector to a high-dimensional space through a fully connected layer, and process it through a nonlinear activation function and a dropout function;

[0187] The classification result generation submodule 052 is used to map the high-dimensional vector back to the low-dimensional space through the fully connected layer, and finally obtain the probability distribution of multiple categories through the Softmax activation function, and take the category with the highest probability as the final detection result.

[0188] It should be noted that the information interaction, execution process, etc. between the modules of the above-mentioned device are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and will not be repeated here.

[0189] Example 3

[0190] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which program code for a multimodal detection method for depression based on a state-space model is stored. The program code includes instructions for executing the multimodal detection method for depression based on a state-space model of embodiment 1 or any possible implementation thereof.

[0191] Computer-readable storage media can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0192] Example 4

[0193] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0194] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the multimodal detection method for depression based on a state-space model of Example 1 or any possible implementation thereof.

[0195] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading software code stored in a memory. The memory can be integrated into the processor or located outside the processor and exist independently.

[0196] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode.

[0197] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing system. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Alternatively, they can be implemented using program code executable by a computing system, and thus, they can be stored in a storage system and executed by the computing system. In some cases, the steps shown or described herein can be performed in a different order than that shown, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0198] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A multimodal depression detection method based on a state-space model, characterized in that: The following steps are involved: Read the subject's facial video sequence and audio sequence as input data through the input module; Using a spatial feature extraction module to extract spatial features from the facial video sequence and the audio sequence respectively to form a video spatial feature vector and an audio spatial feature vector; With the help of a temporal feature learning module, a state space model based on the Mamba architecture is used to perform temporal feature learning on the facial video sequence and the audio sequence respectively to obtain a video temporal feature vector and an audio temporal feature vector; A multimodal fusion module is used to jointly analyze the spatial features and temporal features from the facial video sequence and the audio sequence, first splicing to generate a multimodal spatiotemporal feature vector, and then dynamically weighted feature fusion is achieved through a self-attention mechanism to obtain a fused feature; The output module generates the final depression detection result based on the fusion features after processing through a multi-layer fully connected network.

2. The multimodal detection method for depression based on a state-space model according to claim 1, characterized in that: When the spatial feature extraction module extracts the spatial features, it specifically includes: The facial video sequence is passed through a convolutional neural network to obtain a video feature map, and then a global pooling based on an attention mechanism is performed on the video feature map in the time step dimension to obtain the video spatial feature vector; The audio sequence is passed through a convolutional neural network to obtain an audio feature map, and global pooling based on an attention mechanism is performed on the audio feature map in the time step dimension to obtain the audio spatial feature vector.

3. The multimodal detection method for depression based on a state-space model according to claim 1, characterized in that: When the temporal feature learning module performs temporal feature learning on the facial video sequence, the specific process is as follows: Split the video into T frames and divide each frame into N s ×N s non-overlapping blocks; Initialize the state vector and position encoding vector for each block of each frame; Perform forward and backward scans separately to capture temporal dependencies; The forward and backward states are concatenated to form a final feature, and then global average pooling is performed on all block features to obtain the video temporal feature vector of the facial video sequence.

4. The multimodal detection method for depression based on a state-space model according to claim 1, characterized in that: When the temporal feature learning module performs temporal feature learning on the audio sequence, the specific steps include: Divide the audio signal into multiple frames and apply a Hamming window to the frames at each time step to ensure signal continuity; Mel-frequency cepstral coefficients are used to extract acoustic spectrum features for each frame; Calculate the hidden state of the time step using a time-varying parameter matrix that is dynamically adjusted based on the input data and generate the output feature vector for the current time step; The outputs of all time steps are averaged to obtain the final representation vector of the audio modality.

5. The multimodal detection method for depression based on a state-space model according to claim 1, characterized in that: The specific operations of using the multimodal fusion module to jointly analyze the spatial features and temporal features from the facial video sequence and the audio sequence include: Concatenating the video spatial feature vector with the video temporal feature vector to generate a spatiotemporal embedding of the facial video sequence, and concatenating the audio spatial feature vector with the audio temporal feature vector to generate a spatiotemporal embedding of the audio sequence; splicing the spatiotemporal embedding of the facial video sequence and the spatiotemporal embedding of the audio sequence to generate a multimodal spatiotemporal feature vector; Calculate the query-key-value mapping matrix of the multimodal spatiotemporal feature vector and perform dynamic weighted feature fusion.

6. The multimodal detection method for depression based on a state-space model according to claim 5, characterized in that: The specific steps of generating the final depression detection result through the output module based on the fusion features and processing through a multi-layer fully connected network are as follows: The multimodal spatiotemporal feature vector is mapped to a high-dimensional space through a fully connected layer and processed by a nonlinear activation function and a dropout function; The high-dimensional vector is then mapped back to the low-dimensional space through the fully connected layer, and finally the Softmax activation function is used to obtain the probability distribution of multiple categories, and the category with the highest probability is taken as the final detection result.

7. A multimodal depression detection device based on a state-space model, characterized in that: include: An input module, used for reading the subject's facial video sequence and audio sequence as input data; A spatial feature extraction module, configured to extract spatial features from the facial video sequence and the audio sequence, respectively, to form a video spatial feature vector and an audio spatial feature vector; A temporal feature learning module, configured to perform temporal feature learning on the facial video sequence and the audio sequence respectively using a state space model based on the Mamba architecture, to obtain a video temporal feature vector and an audio temporal feature vector; A multimodal fusion module is used to jointly analyze the spatial features and temporal features from the facial video sequence and the audio sequence, first splicing to generate a multimodal spatiotemporal feature vector, and then performing dynamic weighted feature fusion through a self-attention mechanism to obtain a fused feature; The output module is used to generate a final depression detection result based on the fusion features after processing through a multi-layer fully connected network.

8. The multimodal depression detection device based on the state space model according to claim 7, characterized in that: The spatial feature extraction module includes: A video spatial feature vector generation submodule is used to obtain a video feature map by passing a convolutional neural network through the facial video sequence, and then perform global pooling based on the attention mechanism on the video feature map in the time step dimension to obtain the video spatial feature vector; The audio spatial feature vector generation submodule is used to obtain an audio feature map by passing the audio sequence through a convolutional neural network, and perform global pooling based on the attention mechanism on the audio feature map in the time step dimension to obtain the audio spatial feature vector.

9. The multimodal depression detection device based on the state space model according to claim 7, characterized in that: The temporal feature learning module includes: The video frame and block submodule is used to divide the video into T frame sequences and divide each frame into N s ×N s non-overlapping blocks; The state and position code initialization submodule is used to initialize the state vector and position code vector for each block of each frame; A bidirectional time dependency capture submodule, configured to perform forward scanning and backward scanning to capture time dependency; A feature fusion pooling submodule concatenates the forward and backward states to form a final feature, and then performs global average pooling on all block features to obtain the video temporal feature vector of the facial video sequence; The audio framing and windowing submodule is used to divide the audio signal into multiple frames and apply a Hamming window to the frames at each time step to ensure signal continuity; Acoustic spectrum feature extraction submodule, used to extract acoustic spectrum features for each frame using Mel-frequency cepstral coefficients; The dynamic state update submodule is used to calculate the hidden state of the time step using a time-varying parameter matrix that is dynamically adjusted based on the input data and generate the output feature vector of the current time step; The time series feature averaging submodule is used to average the outputs of all time steps to obtain the final representation vector of the audio modality.

10. The multimodal depression detection device based on the state space model according to claim 9, characterized in that: The multimodal fusion module includes: a video spatiotemporal embedding generation submodule, configured to concatenate the video spatial feature vector and the video temporal feature vector to generate a spatiotemporal embedding of the facial video sequence; An audio spatiotemporal embedding generation submodule, which concatenates the audio spatial feature vector and the audio temporal feature vector to generate a spatiotemporal embedding of the audio sequence; a cross-modal feature splicing submodule, configured to splice the spatiotemporal embedding of the facial video sequence and the spatiotemporal embedding of the audio sequence to generate a multimodal spatiotemporal feature vector; Dynamic weighted fusion submodule, used to calculate the query-key-value mapping matrix of multimodal spatiotemporal feature vectors and perform dynamic weighted feature fusion; The output module includes: The high-dimensional feature extraction submodule is used to map the multimodal spatiotemporal feature vectors to a high-dimensional space through a fully connected layer and process them through a nonlinear activation function and a dropout function; The classification result generation submodule is used to map the high-dimensional vector back to the low-dimensional space through the fully connected layer, and finally obtain the probability distribution of multiple categories through the Softmax activation function, and take the category with the highest probability as the final detection result.

Images