A Depression Detection Method Based on Instruction Fine-tuning Multimodal Speech-Language Model

Abstract

This invention proposes a method for depression detection based on a multimodal speech-language model with instruction fine-tuning. The steps are as follows: First, a multimodal depression instruction dataset is constructed, integrating three types of information—original audio, automatically transcribed text, and emotional descriptions—into structured instruction-response samples. The emotional descriptions are automatically generated using a dedicated large-scale emotional inference model. Next, a low-rank adaptation technique is employed to efficiently fine-tune the parameters of the multimodal speech-language model. Audio features are extracted through an audio encoder and mapped to the text embedding space, then fused with the text and emotional description embeddings in a multimodal manner. The model parameters are optimized using a cross-entropy loss function. Finally, the speech to be detected is input into the fine-tuned model, and the depression recognition result is output. The technical solution provided by this invention enables joint inference based on audio, text, and emotional information, significantly improving the accuracy of depression recognition while offering advantages in parameter efficiency.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, speech signal processing, and mental health detection. Specifically, it relates to a method for detecting depression based on a multimodal speech-language model with instruction fine-tuning. Background Technology

[0002] With increasing attention being paid to mental health issues, early screening and auxiliary diagnostic technologies for depression have become a research hotspot. Among various identification methods, automated detection technology based on speech signals has attracted much attention due to its non-invasiveness, ease of acquisition, and rich emotional and psychological information. Studies have shown that the speech of patients with depression exhibits systematic differences in rhythm, timbre, and speech rate, such as flattened intonation, slower speech rate, and reduced energy. These characteristics provide objective evidence for automated identification.

[0003] However, existing speech-based depression detection technologies still face significant challenges. First, traditional methods largely rely on manually designed acoustic features, such as MFCC and prosodic features. These feature extraction processes are cumbersome and dependent on expert knowledge, resulting in limited generalization ability. While the development of deep learning technology has alleviated feature engineering problems to some extent with end-to-end methods based on deep neural networks, the following shortcomings remain: most studies use only single-modal speech signals, failing to fully utilize the semantic information in the speech content and the paralinguistic emotional cues contained within the speech signal, leading to insufficient information utilization and limiting the upper limit of recognition performance. Second, although some studies have attempted to fuse audio and text information, the fusion methods are mostly simple feature concatenation or decision-level fusion, failing to deeply model the complex cross-modal associations and complementary relationships between audio and text. Furthermore, while large language models have demonstrated powerful capabilities in natural language understanding in recent years, their application in speech emotion and mental health analysis is still in its early stages. Existing methods either simply input speech as text into a large language model, losing the emotional prosodic information in the original speech; or attempt to train dedicated large speech models, but face problems such as scarce training data and high computational costs. More importantly, existing technologies lack a unified framework that can systematically integrate raw audio, text semantics, and sentiment description information, and efficiently utilize the understanding capabilities of large language models in a task-oriented manner.

[0004] In view of this, this invention proposes a depression detection method based on a multimodal speech-language model with instruction fine-tuning, aiming to solve the aforementioned problems. The core idea of ​​this invention is as follows: First, a structured multimodal instruction dataset is constructed, systematically integrating original audio, automatically transcribed text from speech recognition, and automatically generated emotional descriptions from speech to form instruction-response samples that can be understood by large language models. Second, based on this dataset, a parameter-efficient instruction fine-tuning technique is used to adapt a pre-trained multimodal speech-language model, enabling it to simultaneously understand and infer audio, text, and emotional information, thereby achieving more accurate and robust identification of depressive states. Through this series of designs, this invention aims to overcome the shortcomings of existing technologies, significantly enhancing the parameter efficiency and robustness of the model while improving the accuracy of depression identification. Summary of the Invention

[0005] This invention aims to address the following technical problems in existing speech-based depression detection methods: 1) insufficient utilization of single-modal information, failing to systematically integrate audio, text, and emotional information; 2) inefficiency of existing multimodal fusion methods, failing to fully model complex cross-modal relationships; and 3) failure to effectively combine the powerful semantic understanding and reasoning capabilities of large language models, lacking an instruction-based, structured learning framework for depression detection tasks. To solve these technical problems, this invention provides a depression detection method based on a multimodal speech-language model using instruction fine-tuning. This invention achieves efficient and accurate identification of depressive states by constructing a multimodal instruction dataset and employing low-rank adaptive instruction fine-tuning technology.

[0006] The depression detection method based on instruction fine-tuning using a multimodal speech-language model provided by this invention has the following core process: The input speech data is processed by first extracting text transcription using an automatic speech recognition tool and generating speech emotion descriptions using an emotion description generation model, thus constructing a multimodal instruction sample containing the original audio, text transcription, and emotion descriptions. Next, in the model instruction fine-tuning step, an audio encoder is used to extract audio features and map them to the multimodal embedding space. The text transcription, emotion description, and task instructions are concatenated into a text sequence and embedded. The audio embedding and text embedding are then concatenated and input into a pre-trained multimodal speech language model, and low-rank adaptive techniques are used for efficient instruction fine-tuning of parameters. Finally, in the depression detection inference step, the speech to be detected is input into the fine-tuned model, and the depression recognition result is output.

[0007] The technical solution adopted in this invention can be further refined. The specific process of constructing the multimodal instruction dataset mentioned above includes: a) For the first The original audio of each sample Text transcription was obtained through automatic speech recognition tools. ,in , The total number of samples; b) For the first The original audio of each sample Using an emotion description generation model, corresponding speech emotion descriptions of the speaker are generated. ; c) Convert the original audio Text transcription Emotional description With predefined task instructions Integrate and construct a unified multimodal instruction sample and their corresponding response tags Forming command-response pairs .

[0008] The aforementioned model instruction fine-tuning steps specifically include: a) For the first The original audio of each sample Use an audio encoder to extract its frame-level features ,in , For time steps, For feature dimensions; b) Through linear projection matrix Audio features Mapping to audio embedding ,in , For the embedding space dimension of the language model; c) Transcription of text Emotional description With task instructions Concatenate into a unified text sequence This is then converted into a list of individual tokens, which are then converted into the text embedding required by the model. ,in , The length of the text sequence; d) Embed audio With text embedding Concatenate to form a multimodal input sequence = ; e) Using the Low-Rank Adaptation (LoRA) technique, trainable low-rank matrices are inserted into the Transformer layers of the multimodal large speech-language model, keeping most of the pre-trained parameters frozen and only fine-tuning the low-rank matrix parameters. Specifically, for the linear layer weight matrix in the model... By introducing a low-rank decomposition matrix and To adjust the weights, among which It is a low-rank dimension and The adjusted weights are calculated as follows: in and For trainable parameters, Keep frozen, make only minor adjustments and ; f) Use the cross-entropy loss function to perform parameter-efficient fine-tuning of the multimodal large speech language model. The loss function is calculated as follows: in, For the first A true label for each time step. For historical output sequences, These are trainable parameters.

[0009] Compared with existing technologies, the technical solution provided by this invention has significant beneficial effects. First, by constructing a structured multimodal instruction dataset, this invention systematically integrates three types of information: raw audio, text transcription, and emotional description, overcoming the problem of insufficient utilization of modal information in existing technologies and providing the model with a more comprehensive and three-dimensional representation of depression. Second, this invention employs low-rank adaptive instruction fine-tuning technology, which, while efficiently utilizing the powerful semantic understanding capabilities of large language models, significantly reduces the number of training parameters and computational overhead, thereby improving the model's training efficiency and generalization ability. Furthermore, this invention achieves deep fusion of audio, text, and emotional information through multimodal embedding and attention mechanisms, enabling more thorough mining of complementary and correlated information across modalities, thus significantly improving the accuracy and robustness of depression identification. Attached Figure Description

[0010] Figure 1 This is a framework diagram for depression detection based on a multimodal speech and language model with instruction fine-tuning. Detailed Implementation

[0011] To more clearly describe the content of this invention, further explanation is provided below with reference to examples and accompanying drawings. This invention provides a method for detecting depression based on a multimodal speech-language model with instruction fine-tuning. First, the data involved in this embodiment is defined. A multimodal speech sample consists of original audio, text transcription, and emotion description. For the first... 1 sample, whose original audio input is ,in This represents the number of audio sampling points. The corresponding text transcription is... The emotional description is Each sample corresponds to a sentiment label. Where 0 represents non-depression and 1 represents depression, used for model training and evaluation. The specific implementation process of this invention follows the steps described in claim 1, as detailed below:

[0012] 1. The steps for constructing a multimodal instruction dataset include: a) Data preparation: Publicly available audio datasets of depression were used, each containing the original audio files and corresponding text transcriptions; b) Text transcription acquisition: For the first The original audio of each sample Use automatic speech recognition tools to obtain its text transcription Transcription results were manually verified to ensure accuracy. c) Sentiment description generation: For the first The original audio of each sample The SECap emotion description generation model is used to generate corresponding speech emotion descriptions. The generation process uses a fixed prompt: "Based on the given audio, describe the speaker's emotional state in one sentence." d) Instruction Sample Construction: Transcription of Original Audio and Text Emotional description With task instructions Integrate and construct a unified multimodal instruction sample. A specific instruction format example is as follows: System notification: You are a professional psychologist and can determine whether a person is depressed based on the description; User prompt: Audio: <audio> Audio file path< / audio> The speaker in the audio speaks in a sad tone, with a sobbing quality, conveying a sense of grievance and sorrow. The content of the speech is: "I feel extremely painful." Please determine if the speaker is depressed. Response: Depression; The command format can be uniformly expressed as: in Define system roles. This is a task introductory text. The original audio. For text transcription, For emotional description. Response To enhance the diversity and generalization of tasks, task instructions... Employing various semantically equivalent Chinese natural language expressions, for example: "Based on the provided audio, text transcription, and the speaker's emotional state, please determine whether the subject is depressed or not." "Based on audio, text transcription, and the speaker's emotional information, the subjects' states were classified as depressed or non-depressed." In actual construction, one of the predefined equivalent instructions is randomly selected for each sample to avoid the model overfitting to a fixed expression.

[0013] 2. The model instruction fine-tuning steps specifically include: a) For the first The original audio of each sample Frame-level features were extracted using the Whisper-Large-V3 audio encoder. ,in , For time steps, For feature dimensions; b) Through linear projection matrix Audio features Mapping to audio embedding ,in , For the embedding space dimension of the language model; c) Transcription of text Emotional description With task instructions Concatenate into a unified text sequence This is then converted into a list of individual tokens, which are then converted into the text embedding required by the model. ,in , The length of the text sequence; d) Embed audio With text embedding Concatenate to form a multimodal input sequence = ; e) Using the low-rank adaptation technique LoRA, a trainable low-rank matrix is ​​inserted into the Transformer layer of the multimodal large speech language model Qwen2-Audio. Most of the pre-trained parameters are kept frozen, and only the low-rank matrix parameters are fine-tuned. Specifically, for the linear layer weight matrix in the model... By introducing a low-rank decomposition matrix and To adjust the weights, among which It is a low-rank dimension and The adjusted weights are calculated as follows: in and For trainable parameters, Keep frozen, make only minor adjustments and ; f) Use the cross-entropy loss function to perform parameter-efficient fine-tuning of the multimodal large speech language model. The loss function is calculated as follows: in, For the first A true label for each time step. For historical output sequences, These are trainable parameters.

[0014] 3. The reasoning steps for detecting depression, specifically including: After the model is trained, all its parameters are fixed. For a new sample to be tested, the inference process is as follows: First, the original audio of the test sample is input into the trained feature extraction and embedding module to obtain the audio embedding. Simultaneously, text transcription and sentiment description are acquired and embedded as... Secondly, the audio embedding and text embedding are concatenated into a multimodal input sequence. The input is fed into the fine-tuned Qwen2-Audio model. Finally, the model uses multimodal information to reason and outputs whether the sample belongs to "depression" or "non-depression" as the final identification result. Experimental Design

[0015] Experiments were conducted on three publicly available datasets: DAIC, EATD, and CMDC, using 5-fold cross-validation. All speech signals were downsampled to 16kHz, with a batch size of 4 during training, and the AdamW optimizer was used with a learning rate of [missing value]. A total of 20 training cycles were conducted. All experiments were performed on two NVIDIA 4090 GPUs. Experimental results

[0016] To verify the effectiveness of the proposed method, we achieved excellent performance on three datasets: an accuracy of 89.41% and an F1 score of 82.35% on the DAIC dataset; an accuracy of 94.94% and an F1 score of 81.82% on the EATD dataset; and an accuracy of 98.89% and an F1 score of 98.18% on the CMDC dataset. Compared with existing methods, this invention significantly improves both multimodal fusion efficiency and recognition accuracy. Table 1. Metric values ​​of the method of this invention and various comparison algorithms on the DAIC-WoZ dataset. Comparison Methods Accuracy Precision Recall F1 MT-DLR-CombAtt 0.5454 - - 0.4000 FVTC-CNN 0.7350 - - 0.4600 MFCC based RNN 0.7627 - - 0.4600 EmoAudioNet - 0.6600 - - 1DCNN+Trans - 0.7800 0.5800 0.6700 STFN 0.8000 0.6500 0.9200 0.7600 GPT-3.5 - 0.8500 0.8400 0.8200 GPT-4 - 0.7500 0.7600 0.7600 Llama2-13B 0.7200 0.6600 0.6700 Method of the present invention 0.8941 0.8400 0.8077 0.8235

Claims

1. A method for detecting depression based on a multimodal speech-language model with instruction fine-tuning, characterized in that, Includes the following steps: S1. Steps for constructing a multimodal instruction dataset: Collect speech data of depression containing raw audio, text transcription and depression tags; obtain the speaker's speech emotion description through an automatic emotion description generation model; and construct multimodal instruction samples. S2. Model fine-tuning steps: Use a multimodal large speech language model as the base model, map audio features to the text embedding space, and perform multimodal fusion with text and speaker emotion description. Train the model based on instruction fine-tuning technology. S3. Depression detection steps: Input the speech to be detected into the model after fine-tuning in step S2, and output the depression recognition result.

2. The method of claim 1, characterized in that, The multimodal instruction dataset construction step described in step S1 further includes: a) For the first The original audio of each sample Text transcription was obtained through automatic speech recognition tools. ,in , The total number of samples; b) For the first The original audio of each sample Using an emotion description generation model, corresponding speech emotion descriptions of the speaker are generated. ; c) Convert the original audio Text transcription Emotional description With predefined task instructions Integrate and construct a unified multimodal instruction sample and their corresponding response tags Forming command-response pairs .

3. The method according to claim 1, characterized in that, The model fine-tuning in S2 further includes: a) For the first The original audio of each sample Use an audio encoder to extract its frame-level features ,in , For time steps, For feature dimensions; b) Through linear projection matrix Audio features Mapping to audio embedding ,in , For the embedding space dimension of the language model; c) Transcription of text Emotional description With task instructions Concatenate into a unified text sequence This is then converted into a list of individual tokens, which are then converted into the text embedding required by the model. ,in , The length of the text sequence; d) Embed audio With text embedding Concatenate to form a multimodal input sequence = ; e) Using the Low-Rank Adaptation (LoRA) technique, trainable low-rank matrices are inserted into the Transformer layers of the multimodal large speech-language model, keeping most of the pre-trained parameters frozen and only fine-tuning the low-rank matrix parameters. Specifically, for the linear layer weight matrix in the model... By introducing a low-rank decomposition matrix and To adjust the weights, among which It is a low-rank dimension and The adjusted weights are calculated as follows: in and For trainable parameters, Keep frozen, make only minor adjustments and ; f) Use the cross-entropy loss function to perform parameter-efficient fine-tuning of the multimodal large speech language model. The loss function is calculated as follows: in, For the first A true label for each time step. For historical output sequences, These are trainable parameters.

4. The method according to claim 1, characterized in that, In step S3, the depression detection stage, the trained model is loaded, the test subject's voice is input, and the model obtains the depression recognition result through an autoregressive generation method.

Images