A multi-modal depression screening method and device based on video conversation

Abstract

This invention relates to the fields of artificial intelligence and healthcare, and discloses a multimodal depression screening method and device based on video dialogue, comprising the following steps: Step 1: multimodal data acquisition and preprocessing; Step 2: multimodal feature extraction; Step 3: cross-modal feature alignment and fusion; Step 4: depression identification and assessment. In this invention, a progressive multimodal fusion method based on cross-attention is adopted. Visual features are used as queries, and cross-attention calculations are performed with speech features and text features respectively. A fusion representation is obtained through residual connections and layer normalization, enabling effective cross-modal interaction while preserving the specificity of features from different modalities. This solves the problems of poor modal fusion effect and limited discriminative power of fusion representation in existing technologies. Furthermore, this invention introduces a multimodal large model architecture based on ChatGLM, and uses LoRA low-rank adaptation technology for efficient parameter fine-tuning, significantly reducing computational resource requirements.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and healthcare, and in particular to a multimodal depression screening method and device based on video dialogue. Background Technology

[0002] Depression, a common and serious mental disorder, is placing a heavy burden on global public health. According to the World Health Organization, approximately 350 million people worldwide suffer from depression, and its incidence rate is increasing year by year. In recent years, with the development of artificial intelligence technology, automatic depression identification technology based on multimodal data has gradually become a research hotspot. Existing technologies can be mainly divided into the following categories: The first category is identification technology based on single-modal physiological signals, such as electrocardiogram (ECG), electroencephalogram (EEG), and electrodermal conductance. The second category is text analysis-based technology, which identifies depressive tendencies by performing sentiment analysis and semantic mining on text content such as patients' online social platform statements and diaries. The third category is identification technology based on multimodal fusion, which improves identification accuracy by integrating information from multiple modalities such as voice, video, and text.

[0003] However, existing multimodal depression identification technologies still have the following shortcomings: (1) Low efficiency of long-term modeling: The emotional expression of patients with depression during the interview process has significant temporal evolution characteristics. Convolutional neural networks (CNN) are limited by a limited receptive field and have difficulty capturing long-distance dependencies; recurrent neural networks (RNN) have the gradient vanishing problem; although Transformer can capture long-distance dependencies, its computational complexity is high and it is difficult to deploy in resource-constrained scenarios.

[0004] (2) The multimodal fusion effect is not good: Existing technologies often focus on learning modal shared features or modal specific features, but they are not good at capturing cross-modal shared features while retaining modal specific information, resulting in limited discriminative power of fusion representation.

[0005] (3) High cost of data collection and annotation: Depression data involves patient privacy and is difficult to obtain. Annotation requires the participation of professional psychiatrists, which limits the scale of training data and affects model performance.

[0006] (4) Poor adaptability to small sample scenarios: Depression data often suffers from class imbalance, and labeled data is scarce in new scenarios. The recognition performance of existing models drops significantly in cases with few samples.

[0007] Therefore, developing an intelligent screening technology for depression that is efficient, low-cost, easy to apply on a large scale, and has good generalization ability is of great clinical significance and social value. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a multimodal depression screening method and device based on video dialogue, which solves the above problems.

[0009] To achieve the above objectives, the present invention provides a multimodal depression screening method based on video dialogue, comprising the following steps: Step 1: Multimodal Data Acquisition and Preprocessing: Collect multimodal data of the subjects during the video interview, including visual information, speech information and text information; preprocess the multimodal data, wherein the visual data is cropped, scaled, normalized and timestamp aligned, the speech data is denoised, endpoint detected and framed, and the text data is segmented and stop word removed. Step 2: Multimodal Feature Extraction: Feature extraction is performed on the preprocessed multimodal data, including visual feature extraction, speech feature extraction, and text feature extraction. Visual feature extraction uses a face detection model to extract facial keypoint coordinates, and a pre-trained Vision Transformer model to extract facial expression features, combined with a pose estimation model to extract body movement features. Speech feature extraction uses the OpenSMILE model to extract speech prosody features, and a pre-trained HuberT speech model to extract high-level speech sentiment representations. Text feature extraction uses the T5 large language model to extract emotional semantic features from the text. Step 3: Cross-modal feature alignment and fusion: The visual features, speech features, and text features are semantically aligned across modalities. A contrastive learning method is used to map features from different modalities to a unified semantic space. A multimodal fusion network based on cross-attention is used to fuse features. The fusion network uses visual features as queries and performs cross-attention calculations with speech features and text features to obtain the fused multimodal representation. Step 4: Depression Identification and Assessment: Input the fused multimodal representation into a multimodal recognition model based on a large language model, train the model using efficient parameter fine-tuning techniques, and simultaneously optimize the depression binary classification task and the severity assessment task, outputting the depression identification probability and severity assessment results.

[0010] Preferably, the video interview process described in step one is hosted by an AI digital human, using a standardized depression interview questionnaire to guide the subjects to describe their emotional state, daily life, and sleep quality.

[0011] Preferably, the face detection model in step two is the MTCNN model, and the facial key point coordinates are 68 facial key point coordinates; the pose estimation model is the OpenPose model, and the limb movement features are 25 body key point coordinates; the speech prosody features include intonation, speech rate and volume; the features extracted by the large language model T5 are 768-dimensional feature vectors.

[0012] Preferably, the calculation formula for the multimodal fusion network based on cross-attention in step three is as follows: F v→a =CrossAttn( Q = F v , K = F a , V = F a ) F v→t =CrossAttn( Q = F v , K = F t , V = F t ) F fusion =LayerNorm( F v + F v→a + F v→t ) in, Indicates visual characteristics, Representing speech features, Representing text features, This represents the multimodal representation after fusion.

[0013] Preferably, the multimodal recognition model based on a large language model described in step four uses ChatGLM as the base model, and performs efficient parameter fine-tuning using LoRA low-rank adaptation technology; the model training adopts a multi-task learning framework, and the loss function is... L = λ 1 L cls + λ 2 L reg ,in, For depression binary classification loss, For PHQ-9 score regression loss, , .

[0014] Preferably, the model training in step four further includes small sample data augmentation strategies, including: adding noise and speed-changing processing to speech data, performing random cropping, rotation and color jitter processing on video data, performing back-translation and synonym replacement processing on text data; and using a contrastive learning method to construct positive and negative sample pairs and learn cross-modal invariant feature representations.

[0015] Preferably, it also includes step five: result feedback and model update, which specifically involves: feeding back the depression identification and assessment results output by the model to doctors for clinical validation; collecting the bias case data fed back by doctors and adding them to the training dataset; and periodically retraining and optimizing the model.

[0016] A multimodal depression screening device based on video dialogue includes: The multimodal data acquisition module is used to collect visual, audio, and text information from the subjects during the video interview process; The data preprocessing module is used to perform preprocessing operations such as cropping, scaling, normalization, noise reduction, and word segmentation on the multimodal data. The multimodal feature extraction module includes a visual feature extraction unit, a speech feature extraction unit, and a text feature extraction unit, which are used to extract facial expression features, body movement features, speech emotion representation, and text emotion semantic features, respectively. The cross-modal feature alignment and fusion module is used to map features from different modalities to a unified semantic space and to perform feature fusion using a multimodal fusion network based on cross-attention. The depression identification and assessment module is used to input the fused multimodal representation into a multimodal recognition model based on a large language model, and output the depression identification probability and severity assessment results. The results feedback and model update module is used to collect clinical feedback data and iteratively optimize the model.

[0017] Preferably, the visual feature extraction unit includes an MTCNN face detection subunit, a Vision Transformer facial expression feature extraction subunit, and an OpenPose body movement feature extraction subunit; the speech feature extraction unit includes an OpenSMILE prosodic feature extraction subunit and a HuberT high-level speech emotion representation extraction subunit; and the text feature extraction unit includes a T5 large language model text emotion semantic feature extraction subunit.

[0018] Preferably, the depression identification and assessment module includes a ChatGLM-based multimodal large model subunit, a LoRA parameter efficient fine-tuning subunit, a multi-task learning and training subunit, and a small sample data augmentation subunit; the multi-task learning and training subunit simultaneously optimizes the depression binary classification task and the PHQ-9 score regression task.

[0019] This invention provides a multimodal depression screening method and device based on video dialogue. Compared with the prior art, it has the following advantages: 1. In this invention, a progressive multimodal fusion method based on cross-attention is adopted. Visual features are used as queries, and cross-attention calculations are performed with speech features and text features respectively. The fusion representation is obtained through residual connections and layer normalization, so that the features of different modalities can achieve effective cross-modal interaction while retaining their specificity. This solves the problems of poor modal fusion effect and limited discriminative power of fusion representation in the prior art. Secondly, this invention introduces a multimodal large model architecture based on ChatGLM. The parameters are efficiently fine-tuned through LoRA low-rank adaptation technology. Only about 0.3% of the model parameters need to be trained to achieve high-performance depression recognition, which greatly reduces the computational resource requirements and solves the problems of high computational complexity and difficulty in deployment in resource-constrained scenarios of the existing Transformer model. 2. In this invention, a contrastive learning framework is employed for cross-modal semantic alignment. By constructing positive and negative sample pairs and minimizing the InfoNCE loss, features from different modalities are mapped to a unified semantic space, enhancing the model's ability to learn cross-modal invariant features. Simultaneously, combining small-sample data augmentation strategies such as speech noise-adding and speed-changing, random video cropping and rotation, and text back-translation with synonym replacement effectively alleviates the problems of scarce data samples and class imbalance in depression, significantly improving the model's generalization ability in small-sample scenarios. Experimental results show that the method of this invention achieves an accuracy of 87.5% on the binary classification task of depression, a 15% improvement over the best single-modal baseline and an 8.7% improvement over the traditional simple concatenation and fusion method, with an AUC of 0.92. On the PHQ-9 rating prediction task, the RMSE is 3.21 and the PCC is 0.86, significantly outperforming traditional methods (p<0.01), fully validating the technical superiority of this invention. Attached Figure Description

[0020] Figure 1 This is a flowchart of a multimodal depression screening method based on video dialogue proposed in this invention; Figure 2 This is a structural diagram of a multimodal depression screening device based on video dialogue proposed in this invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Please see Figures 1-2 The present invention provides two technical solutions, specifically including the following embodiments: Example 1: A multimodal depression screening method based on video dialogue includes the following steps: Step 1: Multimodal Data Acquisition and Preprocessing: Collect multimodal data from the subjects during the video interview, including visual, audio, and text information; preprocess the multimodal data, including cropping, scaling, normalization, and timestamp alignment of visual data; noise reduction, endpoint detection, and frame segmentation of audio data; and word segmentation and stop word removal of text data. The video interview was hosted by an AI digital human, using a standardized depression interview questionnaire to guide the subjects to describe their emotional state, daily life, and sleep quality. Step 2: Multimodal Feature Extraction: Feature extraction is performed on the preprocessed multimodal data, including visual feature extraction, speech feature extraction, and text feature extraction. Visual feature extraction uses a face detection model to extract facial keypoint coordinates, and a pre-trained Vision Transformer model to extract facial expression features, combined with a pose estimation model to extract body movement features. Speech feature extraction uses the OpenSMILE model to extract speech prosody features, and a pre-trained HuberT speech model to extract high-level speech emotion representations. Text feature extraction uses the T5 large language model to extract emotional semantic features from the text. The face detection model is the MTCNN model, with 68 facial keypoint coordinates; the pose estimation model is the OpenPose model, with 25 body keypoint coordinates; speech prosody features include intonation, speech rate, and volume; the T5 large language model extracts 768-dimensional feature vectors. Step 3: Cross-modal feature alignment and fusion: Visual features, speech features, and text features are semantically aligned across modalities. A contrastive learning method is used to map features from different modalities to a unified semantic space. A cross-attention-based multimodal fusion network is used for feature fusion. The fusion network uses visual features as queries and performs cross-attention calculations with speech and text features to obtain the fused multimodal representation. The calculation formula for the cross-attention-based multimodal fusion network is as follows: Fv→a =CrossAttn( Q = F v , K = F a , V = F a ) F v→t =CrossAttn( Q = F v , K = F t , V = F t ) F fusion =LayerNorm( F v + F v→a + F v→t ) in, Indicates visual characteristics, Representing speech features, Representing text features, This represents the fused multimodal characterization; Step 4: Depression Identification and Assessment: The fused multimodal representations are input into a multimodal recognition model based on a large language model. The model is trained using efficient parameter fine-tuning techniques, simultaneously optimizing the depression binary classification task and the severity assessment task. The output is the depression identification probability and severity assessment results. The multimodal recognition model based on the large language model uses ChatGLM as the base model, and efficient parameter fine-tuning is performed using LoRA low-rank adaptation techniques. Model training employs a multi-task learning framework, with the loss function being... L = λ 1 L cls + λ 2 L reg ,in, For depression binary classification loss, For PHQ-9 score regression loss, , Model training also includes few-sample data augmentation strategies, including: adding noise and speed adjustment to speech data, random cropping, rotation and color jittering to video data, back-translation and synonym replacement to text data; and using contrastive learning methods to construct positive and negative sample pairs and learn cross-modal invariant feature representations.

[0023] The process also includes step five: results feedback and model updates. Specifically, this involves: providing the model's output of depression identification and assessment results to doctors for clinical validation; collecting biased case data from doctors and adding them to the training dataset; periodically retraining and optimizing the model; and collecting feedback results and updating the model after its application, as detailed below: Results Feedback: The depression identification and assessment results output by the model are fed back to the doctor, who then judges and verifies the results based on clinical experience. If the doctor finds a significant discrepancy between the model results and the clinical diagnosis, relevant information should be recorded promptly, including the patient's basic information, interview data, model output results, and the doctor's diagnosis.

[0024] Model Updates: We regularly collect feedback from doctors regarding model identification bias cases, adding this data to the training dataset for retraining and optimization of the multimodal model. By continuously updating the training data and model parameters, the model can better adapt to the characteristics of different patient groups, improving its recognition accuracy and generalization ability. We collaborate with the psychiatry departments of top-tier hospitals, collecting over 50 real clinical interview cases for validation. The model output serves as a supplementary reference; the final diagnosis is confirmed by the doctor. We collect feedback from doctors regarding bias cases and update the model quarterly.

[0025] A multimodal depression screening device based on video dialogue includes: The multimodal data acquisition module is used to collect visual, audio, and text information from the subjects during the video interview process; The data preprocessing module is used to perform preprocessing operations such as cropping, scaling, normalization, noise reduction, and word segmentation on multimodal data. The multimodal feature extraction module includes a visual feature extraction unit, a speech feature extraction unit, and a text feature extraction unit, which are used to extract facial expression features, body movement features, speech emotion representation, and text emotion semantic features, respectively. The visual feature extraction unit includes an MTCNN face detection subunit, a Vision Transformer facial expression feature extraction subunit, and an OpenPose body movement feature extraction subunit; the speech feature extraction unit includes an OpenSMILE prosodic feature extraction subunit and a HuberT high-level speech emotion representation extraction subunit; and the text feature extraction unit includes a T5 large language model text emotion semantic feature extraction subunit. The cross-modal feature alignment and fusion module is used to map features from different modalities to a unified semantic space and uses a multimodal fusion network based on cross-attention to perform feature fusion. The depression identification and assessment module is used to input the fused multimodal representation into a multimodal recognition model based on a large language model, and output the depression identification probability and severity assessment results. The depression identification and assessment module includes a ChatGLM-based multimodal large model sub-unit, a LoRA parameter efficient fine-tuning sub-unit, a multi-task learning and training sub-unit, and a small sample data augmentation sub-unit; the multi-task learning and training sub-unit simultaneously optimizes the depression binary classification task and the PHQ-9 score regression task. The results feedback and model update module is used to collect clinical feedback data and iteratively optimize the model.

[0026] Example 2: Based on Example 1, 80 patients from the outpatient department of a tertiary hospital were selected, including 40 patients diagnosed with depression by two or more associate chief physicians of psychiatry, and 40 healthy controls. There were no statistically significant differences between the two groups in terms of age, gender, and education level (p > 0.05). Data Acquisition: Video interviews were conducted in a quiet, evenly lit consultation room, guided by a trained interviewer using a standardized questionnaire. The interviews lasted 10-15 minutes. Acquisition Equipment: Logitech BRIO 4K camera (1920×1080 resolution, 30fps), Blue Yeti microphone (44.1kHz sampling rate).

[0027] Data annotation: Two attending psychiatrists independently annotated the data, including: ① presence of depression (binary classification); ② PHQ-9 score (0-27). The Kappa coefficient for annotation consistency was 0.85. A third attending physician was used to adjudicate discrepancies in the samples.

[0028] Dataset partitioning: The dataset was divided into a training set (64 cases), a validation set (8 cases), and a test set (8 cases) in an 8:1:1 ratio. Five-fold cross-validation was used to evaluate the model's stability.

[0029] The specific test results are as follows:

[0030] Table 1 Results Analysis: The method of this invention outperforms both the single-modal baseline and traditional fusion methods in all evaluation indicators, with an accuracy improvement of 15% compared to the best single-modal baseline and 8.7% compared to traditional fusion methods. Through the complementary fusion of multimodal information, the accuracy of depression identification is effectively improved.

[0031]

[0032] Table 2 Results analysis: The method of this invention achieved an RMSE of 3.21 on the PHQ-9 score prediction task, which is significantly better than the traditional method (p<0.01), indicating that the model can accurately assess the severity of depression.

[0033] In summary, this invention proposes a multimodal depression screening method and device based on video dialogue. By integrating data from multiple modalities and utilizing advanced multimodal fusion technology and a large language model, it significantly improves the accuracy and efficiency of depression identification, achieving an accuracy improvement of over 8% compared to traditional simple data splicing techniques. This technology has significant innovative value and application potential, and is expected to provide strong support for the early diagnosis and intervention of depression.

[0034] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multimodal depression screening method based on video dialogue, characterized in that: Includes the following steps: Step 1: Multimodal Data Acquisition and Preprocessing: Collect multimodal data from the subjects during the video interview, including visual, audio, and text information; The multimodal data is preprocessed, including cropping, scaling, normalization and timestamp alignment of visual data, noise reduction, endpoint detection and frame segmentation of speech data, and word segmentation and stop word removal of text data. Step 2: Multimodal Feature Extraction: Feature extraction is performed on the preprocessed multimodal data, including visual feature extraction, speech feature extraction, and text feature extraction. Visual feature extraction uses a face detection model to extract facial keypoint coordinates, and a pre-trained Vision Transformer model to extract facial expression features, combined with a pose estimation model to extract body movement features. Speech feature extraction uses the OpenSMILE model to extract speech prosody features, and a pre-trained HuberT speech model to extract high-level speech sentiment representations. Text feature extraction uses the T5 large language model to extract emotional semantic features from the text. Step 3: Cross-modal feature alignment and fusion: The visual features, speech features, and text features are semantically aligned across modalities. A contrastive learning method is used to map features from different modalities to a unified semantic space. A multimodal fusion network based on cross-attention is used to fuse features. The fusion network uses visual features as queries and performs cross-attention calculations with speech features and text features to obtain the fused multimodal representation. Step 4: Depression Identification and Assessment: Input the fused multimodal representation into a multimodal recognition model based on a large language model, train the model using efficient parameter fine-tuning techniques, and simultaneously optimize the depression binary classification task and the severity assessment task, outputting the depression identification probability and severity assessment results.

2. The multimodal depression screening method based on video dialogue according to claim 1, characterized in that: The video interview process described in step one was hosted by an AI digital human, using a standardized depression interview questionnaire to guide the subjects to describe their emotional state, daily life, and sleep quality.

3. The multimodal depression screening method based on video dialogue according to claim 1, characterized in that: The face detection model in step two is the MTCNN model, and the facial key point coordinates are 68 facial key point coordinates; the pose estimation model is the OpenPose model, and the limb movement features are 25 body key point coordinates; the speech prosody features include intonation, speech rate and volume; the features extracted by the large language model T5 are 768-dimensional feature vectors.

4. The multimodal depression screening method based on video dialogue according to claim 1, characterized in that: The calculation formula for the multimodal fusion network based on cross-attention described in step three is as follows: F v→a =CrossAttn( Q = F v , K = F a , V = F a ) F v→t =CrossAttn( Q = F v , K = F t , V = F t ) F fusion =LayerNorm( F v + F v→a + F v→t ) in, Indicates visual characteristics, Representing speech features, Representing text features, This represents the multimodal representation after fusion.

5. The multimodal depression screening method based on video dialogue according to claim 1, characterized in that: The multimodal recognition model based on a large language model described in step four uses ChatGLM as the base model and employs LoRA low-rank adaptation technique for efficient parameter fine-tuning. The model training utilizes a multi-task learning framework, with the loss function being... L = λ 1 L cls + λ 2 L reg ,in, For depression binary classification loss, For PHQ-9 score regression loss, , .

6. The multimodal depression screening method based on video dialogue according to claim 1, characterized in that: The model training described in step four also includes few-sample data augmentation strategies, including: adding noise and speed adjustment to speech data, performing random cropping, rotation and color jittering on video data, back-translation and synonym replacement on text data; and using contrastive learning methods to construct positive and negative sample pairs and learn cross-modal invariant feature representations.

7. The multimodal depression screening method based on video dialogue according to claim 1, characterized in that: It also includes step five: results feedback and model update, which specifically involves feeding back the depression identification and assessment results output by the model to doctors for clinical validation; Collect data on deviation cases reported by doctors and add them to the training dataset. Regularly retrain and optimize the model.

8. A multimodal depression screening device based on video dialogue, based on any one of the multimodal depression screening methods based on video dialogue according to claims 1-7, characterized in that: include: The multimodal data acquisition module is used to collect visual, audio, and text information from the subjects during the video interview process; The data preprocessing module is used to perform preprocessing operations such as cropping, scaling, normalization, noise reduction, and word segmentation on the multimodal data. The multimodal feature extraction module includes a visual feature extraction unit, a speech feature extraction unit, and a text feature extraction unit, which are used to extract facial expression features, body movement features, speech emotion representation, and text emotion semantic features, respectively. The cross-modal feature alignment and fusion module is used to map features from different modalities to a unified semantic space and to perform feature fusion using a multimodal fusion network based on cross-attention. The depression identification and assessment module is used to input the fused multimodal representation into a multimodal recognition model based on a large language model, and output the depression identification probability and severity assessment results. The results feedback and model update module is used to collect clinical feedback data and iteratively optimize the model.

9. A multimodal depression screening device based on video dialogue according to claim 8, characterized in that: The visual feature extraction unit includes an MTCNN face detection subunit, a Vision Transformer facial expression feature extraction subunit, and an OpenPose body movement feature extraction subunit; the speech feature extraction unit includes an OpenSMILE prosodic feature extraction subunit and a HuberT high-level speech emotion representation extraction subunit; the text feature extraction unit includes a T5 large language model text emotion semantic feature extraction subunit.

10. A multimodal depression screening device based on video dialogue according to claim 8, characterized in that: The depression identification and assessment module includes a ChatGLM-based multimodal large model subunit, a LoRA parameter efficient fine-tuning subunit, a multi-task learning and training subunit, and a small sample data augmentation subunit; the multi-task learning and training subunit simultaneously optimizes the depression binary classification task and the PHQ-9 score regression task.

Images