Multi-modal depression detection method and system based on time convolutional neural network

Abstract

The invention provides a multi-mode depression detection method and system based on a time convolutional neural network. The detection method specifically comprises the following steps: constructing a training sample set, wherein the training sample set comprises audios, 3D facial expressions and corresponding text information of depression and non-depression patients; performing 3D facial expression feature extraction on the training sample set to obtain a 3D facial expression feature vector with context awareness; the feature extraction module is used for performing acoustic feature extraction on the audio signals of the training sample set in combination with a Mel-frequency cepstrum coefficient to obtain voice vector features with context awareness; using a Transform model for processing word embedding of the training sample set, and obtaining text features with context awareness; fusing the 3D facial expression features, the voice vector features and the text features to obtain information for depression classification; and substituting the information for depression classification into the time convolutional neural network to obtain depression classification information. According to the invention, the accuracy of depression detection can be improved.

Description

technical field [0001] The invention belongs to the technical field of big data, and in particular relates to a multimodal depression detection method and system based on a temporal convolutional neural network. Background technique [0002] There are nearly 800,000 patients who commit suicide due to depression every year in the world. Compared with other physical diseases, mental disorders are more difficult to detect. In early clinical practice, doctors determined whether a patient was depressed by diagnosing the severity of depressive symptoms during a personal interview. Later, researchers quantitatively analyzed the time-domain characteristics of speech signals, such as pause time, recording time, feedback time to questions, speech speed, etc., to help doctors make auxiliary diagnoses for patients with depression. However, it was found that a single feature is less discriminative for auxiliary clinical diagnosis. In recent years, with the in-depth development of speec...

AI Technical Summary

Problems solved by technology

[0003] In the existing technology, detection methods based on biochemical reagents and EEG are usually used, while technical solutions based on voice, text or images are mostly based on voice data. During clinical interviews, patients may stutter and often Pauses between words, resulting in longer audio, video recordings than non-depressed patients

In short, the existing technology mainly has the following problems: In terms of the amount of training data, most of the existing multimodal depression detection systems based on speech, text or images are trained by limited depression data, so the performance is low ; In terms of feature extraction, the existing feature extraction methods lack the facial expression features of subjects when answering different questions, and their performance in the field of depression detection is insufficient, which limits the performance of the final depression detection system; in terms of depression classification modeling, The existing technology does not consider the long-term dependence of speech, text features and depression diagnosis; in terms of multi-modal fusion, the feature distribution between different modal data is very different. The output of the subsystems is connected in series, because the feature difference between the modalities leads to information loss in the process of feature fusion, so the performance is limited; in terms of model selection, the traditional method mostly uses the depression detection method based on the recurrent neural network, Limit the length of audio and video

Images