Multimodal depression detection method based on voice and text fusion
By employing a multimodal fusion architecture that combines hierarchical feature selection, gated attention fusion, and graph neural network modeling, this approach addresses the issues of insufficient fusion and information suppression in existing multimodal depression detection methods, achieving more efficient depression detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-27
- Publication Date
- 2026-03-13
AI Technical Summary
Existing multimodal depression detection methods suffer from insufficient application of multimodal fusion, lack of specificity in feature extraction level selection, and the risk of information suppression in modality fusion mechanisms, which limits the improvement of detection performance.
A multimodal fusion architecture is adopted, which combines hierarchical feature selection, gated attention fusion, and graph neural network modeling. By dynamically adjusting the fusion of speech and text features through learnable weights, a graph structure is constructed to perform high-order semantic interaction and generate joint representations.
It significantly improves the model's performance in detecting depression, and can adaptively focus on sensitive network layers, dynamically adjust modality contributions, capture deep semantic associations, and improve the accuracy and robustness of detection.
Smart Images
Figure CN121662089A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of depression detection, and more particularly to a multimodal depression detection method based on the fusion of speech and text. Background Technology
[0002] Depression is a common mental disorder affecting the physical and mental health of hundreds of millions of people worldwide and is the leading cause of disability globally. Early and accurate diagnosis and targeted treatment of depression are essential. Traditional diagnostic methods primarily rely on clinical interviews and psychological scale assessments, which suffer from high subjectivity, high costs, and poor accessibility. With the development of artificial intelligence and computational psychiatry, automated depression detection technologies based on biomarkers have gradually become a research hotspot. Among these, speech, as a non-invasive, easily collected physiological signal rich in emotional and cognitive state information, has been proven to reflect abnormal changes in vocal mechanisms, intonation patterns, and speech fluency in individuals under depressive states, making it suitable as a remote, continuous, and objective assessment medium. Simultaneously, textual features such as vocabulary selection, syntactic structure, semantic coherence, and emotional tendency in an individual's language expression can also effectively reveal patterns of change in their psychological state.
[0003] In existing technologies, research has attempted to combine speech and text modalities for depression detection. This paper proposes a speech and text-based depression detection and depression severity assessment system. The system collects and preprocesses recorded data through a speech acquisition module, then uses a local feature acquisition module to encode features from both speech and text data, obtaining their respective local feature representations. Regarding the model architecture, a W-VALD+GRU (Gated RecurrentUnit) model is used for pre-training of the speech monomodal model, while an ELMo (Embeddings from Language Models) + BiLSTM (Bidirectional Long Short-Term Memory) network is used for pre-training of the text monomodal model to optimize the feature extraction process. Finally, the automatic detection and assessment module, through an interactive attention fusion and tensor fusion multimodal fusion module, combines a multimodal classification layer and a regression layer to achieve automatic detection of depression and automatic assessment of depression severity.
[0004] In addition, there is a bimodal depression detection method based on Wav2Vec 2.0 (Waveform to Vector) and BERT (Bidirectional Encoder Representations from Transformers). This method uses multi-scale convolutional networks and bidirectional long short-term memory networks to extract deep features from speech and text respectively, and integrates the multimodal features through an adaptive pooling mechanism to complete the classification task of depressive states within a unified framework.
[0005] Although some progress has been made in recent years in multimodal automatic depression detection methods based on voice and text, existing technologies still have several key limitations, mainly in the following aspects: Multimodal fusion applications are still insufficient: Most current automatic depression identification methods still rely primarily on single-modal analysis (such as speech or text only), depending on a single information source for modeling. Although some studies have attempted to introduce bimodal fusion strategies of speech and text, such methods are still in the initial exploratory stage in the overall technical approach and have not yet formed a systematic and efficient multimodal collaborative analysis framework. This makes it difficult for models to fully explore and utilize complementary information across modalities, limiting further improvements in detection performance.
[0006] The selection of feature extraction layers lacks specificity: Existing methods based on deep pre-trained models (such as Wav2Vec 2.0, BERT, ELMo, etc.) typically only use the last hidden state of the model output as semantic representation for subsequent classification tasks. However, acoustic and linguistic abnormalities related to depression may exhibit differentiated sensitivities across different network layers. Simply using the last layer's features may lead to the loss of key biological behavioral markers related to depression, weakening the model's discriminative ability.
[0007] Modality fusion mechanisms carry the risk of information suppression: Current mainstream multimodal fusion methods mostly adopt a sequential fusion strategy of "concatenation followed by weighting," that is, first concatenating the vectors of speech and text modal features, and then processing the fused joint representation through attention mechanisms or other weight allocation methods. Such methods fail to fully consider the semantic integrity and contribution differences of each modality during the fusion process, which can easily lead to the problem of stronger modalities dominating and weaker modalities being submerged. Summary of the Invention
[0008] The purpose of this invention is to address the shortcomings of existing technologies by proposing a multimodal depression detection method based on the fusion of speech and text.
[0009] The objective of this invention is achieved through the following technical solution: a multimodal depression detection method based on speech and text fusion, the method comprising: S1. Receive the user's voice signal and preprocess the voice signal and the corresponding transcribed text; S2. Extract speech features and text features from the speech signal and transcribed text respectively. During the extraction process, select key layers and use learnable weights for weighted fusion. S3. Through a learnable vector gating function, dynamically adjust the fusion weights of speech and text information on each feature channel according to the quality and semantic consistency of the input modality. S4. Construct speech and text modalities as graph nodes, and use graph neural networks to model interaction patterns to obtain joint representations; S5. Input the joint representation into the classifier to obtain the depression detection results.
[0010] Furthermore, the extraction of speech features and text features specifically includes: Using BERT hidden unit as the speech encoder, the hidden state sequence of each layer is obtained, and a set of key layers are selected and weighted and fused through a learnable scalar to obtain the speech features after layer selection. Using BERT as the text encoder, the hidden state sequence of each layer is obtained. A set of key layers are selected and weighted and fused using a learnable scalar to obtain the text features after layer selection.
[0011] Furthermore, the dynamic adjustment of the fusion weights of speech and text information on each feature channel specifically involves: calculating the cross-attention of text to speech to obtain the speech features enhanced by cross-attention. Calculate the cross-attention between speech and text to obtain the text features enhanced by cross-attention. Introducing gated functions G The intensity of information flow is controlled within the range [0, 1]. in, σ For the sigmoid function, [⋅;⋅] represents vector concatenation. For a learnable fully connected weight matrix, Let be a learnable bias vector, and ⊙ represent element-wise multiplication. These are the features after fusion.
[0012] Furthermore, the specific steps of constructing the bimodal features into graph features are as follows: Construct a fully connected graph containing two nodes. , where: node set , representing speech and text modalities respectively; initial node features Edge set This indicates bidirectional interaction, with edge weights determined by the cross-attention score.
[0013] Furthermore, the specific use of graph neural networks to model interaction patterns includes: in Attention coefficient: The node representation is enhanced after the update. and Final joint characterization: .
[0014] On the other hand, the specification also provides a system for implementing the method, the system comprising: The system consists of a preprocessing module, a bimodal coding and hierarchical feature extraction module, a gated attention fusion module, a cross-modal graph interaction modeling module, and a classification decision module. The system receives the user's voice signal and the corresponding transcribed text as input. After preprocessing by the preprocessing module, the data is fed into the voice coding branch and text coding branch of the dual-modal coding and hierarchical feature extraction module to extract multi-level semantic representations. Subsequently, the data is initially fused through the gated attention fusion module, and then subjected to high-order semantic interaction modeling of graph neural networks in the cross-modal graph interaction modeling module. Finally, the data is input to the classification decision module to output the depression classification result.
[0015] Furthermore, the system's training process includes: training the trainable parameters in the system using end-to-end and two-stage joint fine-tuning, specifically: Phase 1: Freeze Hubert and the BERT backbone, and train only the gating attention fusion module and the classification decision module; Phase 2: Unfreeze some high-level parameters of the pre-trained model, perform end-to-end joint fine-tuning, and optimize the overall network using the cross-entropy loss function.
[0016] On the other hand, a multimodal depression detection device based on speech and text fusion is also provided, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the aforementioned multimodal depression detection method based on speech and text fusion.
[0017] On the other hand, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements the aforementioned multimodal depression detection method based on speech and text fusion.
[0018] The beneficial effects of this invention are: A hierarchical feature aggregation mechanism based on learnable weights is proposed. This mechanism dynamically selects the features with the strongest depression recognition capabilities from multiple intermediate layers of HubERT and BERT, and then performs weighted fusion using trainable weights normalized by softmax. This mechanism enables the model to adaptively focus on the network layers most sensitive to depression, fully exploring multi-granular pathological cues in the deep model.
[0019] A gated attention fusion module is designed, combining bidirectional cross-attention with a vector-based gating function to achieve fine-grained, channel-level modal balance. The gating value can dynamically adjust the contribution ratio of speech and text based on different feature dimensions.
[0020] For the first time, speech and text modalities are modeled as two nodes in a graph structure, and a graph neural network (GNN) is used to explicitly model the high-order semantic relationships between them. Message passing is achieved through a graph attention mechanism to capture deep collaborative patterns between modalities and generate joint representations with greater semantic consistency. Attached Figure Description
[0021] Figure 1 A system architecture diagram provided for embodiments of the present invention; Figure 2 This is a structural diagram of the dual-modal coding and hierarchical feature extraction module provided in an embodiment of the present invention; Figure 3 This is a structural diagram of the gated attention fusion module provided in an embodiment of the present invention; Figure 4 This is a structural diagram of the cross-modal graph interaction modeling module provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the apparatus provided in an embodiment of the present invention. Detailed Implementation
[0022] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0023] like Figure 1 As shown, this invention provides a multimodal depression detection method based on speech and text fusion, which differs from existing technologies: traditional methods typically use single-layer feature concatenation followed by direct input into a classifier, or use simple attention weighting. This invention introduces for the first time a three-level progressive fusion architecture of "hierarchical feature selection + gating fusion + graph structure interaction," significantly enhancing the model's ability to model complex cross-modal dependencies. Specific steps include: S1. Receive the user's voice signal and preprocess the voice signal and the corresponding transcribed text; S2. Extract speech features and text features from the speech signal and transcribed text respectively. During the extraction process, select key layers and use learnable weights for weighted fusion. Multi-level features are extracted from both speech and text modalities, and the most discriminative intermediate layer representations are aggregated using learnable weights, thus overcoming the limitation of "using only top-level features." Figure 2 As shown.
[0024] Speech Feature Extraction: Hierarchical Feature Selection Based on HuberT HuBERT (Hidden-unit BERT) is used as the speech encoder. Given a speech signal... After being transformed by the Mel spectrogram, it is input into the HuberT model. L Hidden state sequence of the layer Transformer encoder: in Indicates the first l The hidden feature matrix of the layer, d For feature dimension, T l This is the time step for this layer.
[0025] Introducing a hierarchical selection strategy: not using the last level Instead, select a set of key layers (initially layers 6, 9, and 12, and then perform a sliding window operation). And through learnable scalars Perform weighted fusion: Where Pool(⋅) represents the average pooling operation over time. For the final global representation of speech, Normalization via softmax: all It is used as a trainable parameter in backpropagation optimization.
[0026] Text Feature Extraction: Hierarchical Feature Selection Based on BERT BERT (Bidirectional Encoder Representations from Transformers) was used as the text encoder.
[0027] Given transcribed text x t After word segmentation, the input is given to BERT to obtain... Hidden layer state: Similarly, select the key layer set S t And perform weighted fusion: Where CLS(⋅) represents retrieving the vector corresponding to the classification label of each layer. These are learnable weights (normalized by softmax).
[0028] Technical explanation: This hierarchical feature selection strategy allows the model to automatically learn which network layers contribute most to depression identification.
[0029] Unlike existing technologies that use ELMo+BiLSTM to extract features from a fixed layer, or use BERT but only extract the last layer, this invention uses a learnable hierarchical weighting mechanism to dynamically focus on the semantic layers most conducive to depression recognition, thereby improving feature sensitivity.
[0030] S3. Through a learnable vector gating function, dynamically adjust the fusion weights of speech and text information on each feature channel according to the quality and semantic consistency of the input modality. After completing bimodal independent encoding, by introducing a gating mechanism and bidirectional attention, preliminary interaction and adaptive weight adjustment between speech and text modalities are achieved, avoiding the modal suppression problem caused by the traditional "concatenation followed by weighting" method. Figure 3 As shown.
[0031] Let the speech representation be The text is represented as .
[0032] Bidirectional cross-attention calculation: Computational text focuses on speech: The computational focus of speech on text: in This is the learnable parameter matrix.
[0033] Gating fusion mechanism Introducing gating functions G The intensity of information flow is controlled within the range [0, 1]. in σ For the sigmoid function, [⋅;⋅] represents vector concatenation. For a learnable fully connected weight matrix, Let be a learnable bias vector, and ⊙ represent element-wise multiplication. These are the features after fusion.
[0034] Technical Function Description: This gating mechanism allows the model to dynamically adjust modal contributions based on input quality. When speech is ambiguous, the speech representation... It will be distorted. It will become unreliable; the gating function will trigger a push after capturing the above information. G Decrease, Increase The system relies more on text information; conversely, it enhances speech weight to achieve adaptive and fair modal fusion.
[0035] Unlike existing technologies that use interactive attention fusion and tensor fusion, or simple concatenation and adaptive pooling mechanisms, this module employs interactive attention + gating modulation to achieve bidirectional semantic guidance and avoid one-way information overload.
[0036] S4. Construct speech and text modalities as graph nodes, and use graph neural networks to model interaction patterns to obtain joint representations; To further explore higher-order semantic dependencies between speech and text, this module constructs a graph structure from bimodal features and utilizes a Graph Neural Network (GNN) to model complex interaction patterns, such as... Figure 4 As shown.
[0037] Graph structure construction: Construct a fully connected graph containing two nodes. ,in: Node set , representing speech and text modalities respectively; Initial node features Edge set , indicating two-way interaction.
[0038] Edge weights are determined by the cross-attention score: Graph convolution update: Message passing is performed using a Graph Attention Network (GAT). in The weight matrix is a learnable matrix. Initial features for speech or text nodes ( ), For neighbors of voice or text nodes, i and j The value is v or t , Attention coefficient: in a The attention vector is updated to obtain the enhanced node representation. and .
[0039] Final joint characterization: Technical Function Explanation: GNN can explicitly model nonlinear and higher-order correlations between modalities, capture "combined" depression biomarkers, and improve model interpretability and robustness.
[0040] Unlike existing technologies, which lack the ability to model cross-modal relationships in a structured way, this invention models bimodal interactions as a graph structure and uses GNNs to achieve deep semantic fusion.
[0041] S5. Input the joint representation into the classifier to obtain the depression detection results.
[0042] Joint representation of S4 output Input fully connected classifier: Output the probability of a depressive state .
[0043] Training strategy: End-to-end and two-stage joint fine-tuning Phase 1 (optional): Freeze HuBERT and the BERT backbone, and train only the fusion module and classifier (two-stage training). Phase Two: Unfreeze some high-level parameters of the pre-trained model, perform end-to-end joint fine-tuning, and optimize the overall network using the cross-entropy loss function: in For sample labels, The probability values output by the classification decision module.
[0044] like Figure 1 As shown, the multimodal depression detection system proposed in this invention consists of the following five main functional modules: data preprocessing module, bimodal coding and hierarchical feature extraction module, gated attention fusion module, cross-modal graph interaction modeling module, and classification decision module.
[0045] The system receives the user's voice signal and the corresponding transcribed text as input. After preprocessing by the preprocessing module, the data is fed into the voice coding branch and text coding branch of the dual-modal coding and hierarchical feature extraction module to extract multi-level semantic representations. Subsequently, the data is initially fused through the gated attention fusion module, and then subjected to high-order semantic interaction modeling of graph neural networks in the cross-modal graph interaction modeling module. Finally, the data is input to the classification decision module to output the depression classification result (e.g., "depressed patient" / "normal population").
[0046] Corresponding to the aforementioned embodiment of a multimodal depression detection method based on speech and text fusion, the present invention also provides an embodiment of a multimodal depression detection device based on speech and text fusion.
[0047] See Figure 5 The present invention provides a multimodal depression detection device based on speech and text fusion, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a multimodal depression detection method based on speech and text fusion as described in the above embodiment.
[0048] The embodiment of the multimodal depression detection device based on voice and text fusion provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device reading the corresponding computer program instructions from non-volatile memory into memory and executing them. From a hardware perspective, such as... Figure 5 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the multimodal depression detection device based on speech and text fusion provided by this invention. (Except for...) Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.
[0049] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0050] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0051] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a multimodal depression detection method based on speech and text fusion as described in the above embodiments.
[0052] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.
[0053] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned multimodal depression detection method based on speech and text fusion.
[0054] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.
[0055] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A multimodal depression detection method based on speech and text fusion, characterized in that, The method includes: S1. Receive the user's voice signal and preprocess the voice signal and the corresponding transcribed text; S2. Extract speech features and text features from the speech signal and transcribed text respectively. During the extraction process, select key layers and use learnable weights for weighted fusion. S3. Through a learnable vector gating function, dynamically adjust the fusion weights of speech and text information on each feature channel according to the quality and semantic consistency of the input modality. S4. Construct speech and text modalities as graph nodes, and use graph neural networks to model interaction patterns to obtain joint representations; S5. Input the joint representation into the classifier to obtain the depression detection results.
2. The multimodal depression detection method based on speech and text fusion according to claim 1, characterized in that, The extraction of speech features and text features specifically includes: Using BERT hidden unit as the speech encoder, the hidden state sequence of each layer is obtained, and a set of key layers are selected and weighted and fused through a learnable scalar to obtain the speech features after layer selection. Using BERT as the text encoder, the hidden state sequence of each layer is obtained. A set of key layers are selected and weighted and fused using a learnable scalar to obtain the text features after layer selection.
3. The multimodal depression detection method based on speech and text fusion according to claim 1, characterized in that, The specific method for dynamically adjusting the fusion weights of speech and text information on each feature channel is as follows: calculate the cross-attention of text to speech to obtain the speech features enhanced by cross-attention. Calculate the cross-attention between speech and text to obtain the text features enhanced by cross-attention. Introducing gating functions G The intensity of information flow is controlled within the range [0, 1]. in, σ For the sigmoid function, [⋅;⋅] represents vector concatenation. For a learnable fully connected weight matrix, Let be a learnable bias vector, and ⊙ represent element-wise multiplication. These are the features after fusion.
4. The multimodal depression detection method based on speech and text fusion according to claim 1, characterized in that, The specific steps of constructing speech and text modalities into graph nodes are as follows: Construct a fully connected graph containing two nodes. , where: node set , representing speech and text modalities respectively; initial node features Edge set This indicates bidirectional interaction, with edge weights determined by the cross-attention score.
5. The multimodal depression detection method based on speech and text fusion according to claim 1, characterized in that, The specific features of using graph neural networks to model interaction modes include: in The weight matrix is a learnable matrix. The initial features for speech or text nodes. For neighbors of voice or text nodes, i and j The value is v or t， Attention coefficient: in, a The attention vector is updated to obtain the enhanced node representation. and Final joint characterization: 。 6. A system for implementing the method according to any one of claims 1-5, characterized in that, The system includes: The module includes a data preprocessing module, a bimodal coding and hierarchical feature extraction module, a gated attention fusion module, a cross-modal graph interaction modeling module, and a classification decision module. The system receives the user's voice signal and the corresponding transcribed text as input. After preprocessing by the preprocessing module, the data is fed into the voice coding branch and text coding branch of the dual-modal coding and hierarchical feature extraction module to extract multi-level semantic representations. Subsequently, the data is initially fused through the gated attention fusion module, and then subjected to high-order semantic interaction modeling of graph neural networks in the cross-modal graph interaction modeling module. Finally, the data is input to the classification decision module to output the depression classification result.
7. The system according to claim 6, characterized in that, The system's training process includes: training the trainable parameters in the system using end-to-end and two-stage joint fine-tuning, specifically: Phase 1: Freeze Hubert and the BERT backbone, and train only the gating attention fusion module and the classification decision module; Phase 2: Unfreeze some high-level parameters of the pre-trained model, perform end-to-end joint fine-tuning, and optimize the overall network using the cross-entropy loss function.
8. A multimodal depression detection device based on speech and text fusion, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a multimodal depression detection method based on speech and text fusion as described in any one of claims 1-7.
9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a multimodal depression detection method based on speech and text fusion as described in any one of claims 1-7.