A method for auxiliary detection of depression in the population based on multimodal deep neural network

Abstract

This invention belongs to the field of computer natural language processing sentiment analysis and proposes a method for auxiliary detection of depression in the population based on a multimodal deep neural network. First, audio, text, and video modal data are collected and read; high-dimensional features are extracted from the three modal data through three branch networks. The audio modality uses Wav2vec2.0 to encode the audio data, and the feature extraction module network extracts the audio high-dimensional features; the text modality uses Bert to encode the text data, and a bidirectional long-short time memory network extracts the text high-dimensional features; the coordinate information of key points of the face and head is obtained from the video modality, and the high-dimensional features of the video are extracted through a time-distributed convolutional neural network; by building a self-attention module, the three high-dimensional features are fully integrated to improve the accuracy of depression detection; by using a weighted cross-entropy loss function, the influence of interference noise on the network learning ability is suppressed, thereby accurately predicting depression.

Description

Technical Field

[0001] The present invention relates to the field of computer natural language processing sentiment analysis, and in particular to a method for auxiliary detection of depression in a population based on a multimodal deep neural network. Background Art

[0002] Depression is a common mental illness that has affected millions of people worldwide for decades. Depression can lead to loss of sleep, concentration, and well-being, severely impairing quality of life. In extreme cases, it can even lead to suicide. Furthermore, depression is a leading cause of disability. From a socioeconomic perspective, people with depression lose four additional hours of work per week compared to healthy individuals, potentially impacting both individual lives and overall economic productivity.

[0003] Therefore, it is very necessary to be able to extract and diagnose depression to alleviate these adverse consequences for society and individuals. In Min Baoquan, Zhou Aihong, Liang Feng, et al. Clinical application of Patient Health Questionnaire Depression Scale (PHQ-9) [J]. Neurological Diseases and Mental Health, 2013(006):569-572., Kurt Kroenke, Tara W. Strine, Robert L. Spitzer, Janet BW Williams, Joyce T. Berry, Ali H. Mokdad, The PHQ-8 as a measure of current depression in the general population, Journal of Affective Disorders, Volume 114, Issues 1–3, 2009, Pages 163-173 and Bernd Kurt Kroenke, Kerstin Detecting and Monitoring Depression with a Two-Item Questionnaire (PHQ-2), Journal of Psychosomatic Research, Volume 58, Issue 2, 2005, Pages 163-171, mentions that current methods for diagnosing depression include clinical diagnosis through face-to-face interviews with psychiatrists, as well as questionnaires such as the PHQ-2, PHQ-8, and PHQ-9. While diagnosis by a psychiatrist can more accurately detect disease manifestations and symptoms, the limited availability of psychiatrists limits in-person interviews, making this approach less widely applicable. Compared to clinical diagnosis, the Patient Health Questionnaire (PHQ-8) has lower sensitivity and specificity, making it less practical for diagnosing depression. For example, the PHQ-8 has a sensitivity of 77% and a specificity of 62%, significantly lower than the structured clinical interview's sensitivity of 95% and specificity of 84%. Furthermore, the pathogenesis of depression is still under investigation, making it difficult for doctors to diagnose and treat depression, especially in the early stages of clinical development. With the rapid development of artificial intelligence, understanding the relationship between depression and daily behavior can help diagnose depression early, effectively reducing labor costs, clinical misdiagnosis, and doctors' experience. In addition, mental health testing and cloud-based remote diagnosis can be achieved through automated depression diagnosis systems, which provides an opportunity to apply automatic speech recognition technology to diagnose depression. This method not only reduces dependence on human resources, but is also more scalable than in-person screening. Over the past decade, many technologies have been developed to detect various mental disorders, including anxiety, PTSD, and Alzheimer's disease.

[0004] Compared with ordinary people, patients with depression usually have different facial expressions and daily communication situations, such as dull expressions, abnormal head movements, low voice tones, and relatively negative text semantics. Due to the characteristics reflected by these different modalities, the detection of depressive states using multimodal combination is more accurate than that using a single modality. Summary of the Invention

[0005] This paper aims to propose a method for assisting the detection of depression in people based on a multimodal deep neural network. By extracting high-dimensional features from easily captured daily behaviors such as speech, text transcriptions, and video footage, this method can accurately detect depression, protecting the subject's privacy while also assisting in symptom detection.

[0006] The technical solution of the present invention is:

[0007] A method for auxiliary detection of depression in the population based on a multimodal deep neural network, comprising the following steps:

[0008] Step 1): Extract high-dimensional features of the depressive state using a feature extraction module based on the audio modality;

[0009] The feature extraction module is composed of a multi-head self-attention network (SSA), which is mainly composed of a multi-head self-attention mechanism structure and a stacked feedforward neural network. The feedforward neural network includes two fully connected layers. The activation function of the first fully connected layer is ReLU, and the activation function of the second fully connected layer is a linear activation function.

[0010] Step 2): Extract high-dimensional features of depression status using a bidirectional long-short-term memory network framework model based on text modality;

[0011] The bidirectional long short-term memory network is a BiLSTM network that combines bidirectional LSTMs. The BiLSTM network uses two LSTM layers. The LSTM model can effectively capture long-range dependencies because the LSTM learns what information to remember and what to forget through training. In traditional recurrent neural network and LSTM models, information can only propagate forward, so the state at time t depends only on the text information before time t. To ensure that each moment contains more complete contextual information, the BiLSTM model, consisting of LSTM neurons and a bidirectional recurrent neural network (BiRNN) model, can be used to capture contextual information. The BiLSTM is an extension of the LSTM model.

[0012] Step 3): Extract high-dimensional features of depression state using a time-distributed convolutional neural network model based on the video modality;

[0013] The temporal distributed convolutional neural network model consists of two parallel branches. The T-CNN branch has a multi-layer structure, including five dilated convolution blocks and four maximum pooling layers. The dilated convolution blocks are used to obtain information within different perception ranges. The maximum pooling layer is used to reduce feature resolution and extract high-dimensional features.

[0014] Step 4): Fuse the high-dimensional features of depression status extracted from the three modalities, output binary labels through the self-attention module, and detect depression status in the crowd;

[0015] By learning the features of the three modalities, we can classify the presence or absence of depression. There are only two prediction results, with the predicted probabilities p and 1-p respectively. We use the cross entropy loss function, which is defined as follows:

[0016]

[0017] l ce =-[y·log(p)+(1-y)·log(1-p)] (1)

[0018] Where m is the modal channel used, l ce is the cross entropy loss function defined, x m is the multimodal feature vector of three-channel splicing, ω m is the relative x in the self-attention network m The learned weights, y, represent the label of the sample, 1 for depression and 0 for no depression, and p represents the probability that the sample is predicted to have depression.

[0019] In the step 1), the specific steps include:

[0020] Step 1.1): Audio sequence encoding

[0021] A pre-trained Wav2vec2.0 model is used as the Wav2vec2.0 audio encoding module to extract audio clips. Bidirectional Encoder Representations from Transformers (BERT) is a pre-training method for natural language processing proposed by Google. It is a bidirectional superposition of multiple Transformers. It trains a general "language understanding" model on a large text corpus and then applies this model to various downstream tasks.

[0022] The audio clips of the subjects were recorded through video recording. After preprocessing with noise reduction, impurity removal, and mute removal, they were input into the Wav2vec2.0 audio encoding module, with the output feature dimension being 512 × the number of frames. The Wav2vec2.0 audio encoding module includes the vq-wav2vec quantization module and the Transformer structure.

[0023] Step 1.2): Feature extraction module performs feature extraction

[0024] The features output by the Wav2vec2.0 audio encoding module Input to the feature extraction module, where T and d m are the length and dimension of the input respectively; the feature extraction module projects x three times to obtain the query vector Q, key vector K and value vector V, and then splits Q, K and V into h parts to generate d h dimensional features, among which h is a positive number. The features of each part are output after passing through a multi-head self-attention mechanism structure. Q' is output. At the same time, the input x is input into the feedforward neural network after passing through another multi-head self-attention mechanism structure to obtain a weighted feature vector and map it to K' and V'. Q', K' and V' are input into the next multi-head self-attention structure and output to the last layer of the feedforward neural network to obtain the output features of each part. The outputs of h dimensions are connected in series to form the final output. The mapping formula of the feature extraction module is as follows:

[0025]

[0026] K′,V′=max(0,Q′W1+b1)W2+b2 (3)

[0027]

[0028] Where W1 and W2 are learnable weights, and b1 and b2 are biases.

[0029] In the step 2), the specific steps include:

[0030] 2.1): Text sequence encoding

[0031] The audio of the person being tested is transcribed to obtain text data, which is then cleaned and input into the Bert text encoding module. The Bert text encoding module is a pre-trained Bert model. The input is the transcribed text of each sample and the output is a one-dimensional feature vector of 1×768.

[0032] 2.2): Bidirectional long short-term memory network extracts high-dimensional features of text

[0033] The one-dimensional feature vector obtained in step 2.1) is input into the bidirectional long short-term memory network. The one-dimensional feature vector is first input into the front LSTM layer, and then the reverse time series is input into the back LSTM layer.

[0034] In the step 3), the specific steps include:

[0035] 3.1) Collecting the 3D coordinate information of the subject's head movements during the interview by wearing sensors, and collecting the 3D coordinate information of the subject's facial key points during the interview by recording video;

[0036] The obtained head movement and three-dimensional coordinate information of key points are read and obtained The feature vector of , where T is the time step and D is the feature dimension;

[0037] 3.2) Using the 3D coordinate feature vector sequence of facial key points and the 3D coordinate feature vector sequence of head movements as input, a time-distributed convolutional neural network is used to extract high-dimensional visual features.

[0038] Input the facial key point 3D coordinate feature vector sequence and the head movement 3D coordinate feature vector sequence to two parallel T-CNN branches for feature learning;

[0039] enter The convolution operation of the T-CNN branch is expressed as:

[0040]

[0041] Where d is the dilation factor, k is the convolution kernel size, and b is the bias. Zero padding is used to maintain the same input and output shapes for the T-CNN. The dilation factor is a multiple of 2 to capture temporal information at different time spans along the path. The two parallel paths are connected by an activation function.

[0042] In the step 4), the specific steps include:

[0043] 4.1) Integrate the high-dimensional features extracted from the three branch models to obtain vector I, which is input into the self-attention module. When processing these high-dimensional feature vector sequences, the individual vectors are not independent. Since the depression-related features are extracted from different modalities, the correlations between vectors from different modalities are also different, so it is necessary to establish non-local dependencies. The self-attention module fuses the learned features and assigns weights to different feature channels, thus obtaining a weight matrix representing the importance of different features from the three modalities.

[0044] When the input vector I is input, the calculation process of the self-attention mechanism is:

[0045] Calculate the three vector matrices of the fused vector I, namely Q F , K F , V F matrix:

[0046] Q F =W q I (2)

[0047] K F =W k I (3)

[0048] V F =W v I (4)

[0049] Calculate the attention score matrix A′

[0050] A=K T Q F(5)

[0051] A′=softmax(A) (6)

[0052] Calculate the output matrix O

[0053] O=V F A′ (7)

[0054] Where W q , W k , W v is a learnable parameter.

[0055] In step 2.1), when the text is input into the Bert text encoding module, it is first encoded into a code that can be recognized by the model computer language; first, the text is encoded into numbers according to the dictionary, which is a token embedding; when the input text is two sentences, it is separated by the [SEP] mark to obtain a segment embedding, and the element corresponding to the first sentence is encoded as 0, and the element corresponding to the second sentence is encoded as 1; when the input text element is position information, it is a position embedding; the three embeddings are combined and input into the Bert text encoding module.

[0056] Beneficial effects of the present invention:

[0057] 1) Using three different branch networks to extract high-dimensional features of depression from audio, text, and video modalities, and then performing deep fusion to detect depression;

[0058] 2) To obtain abstract video modal features, a multi-scale temporal distribution convolutional neural network is constructed to improve the high-dimensional depressive state characteristics in the modality;

[0059] 3) Propose a three-channel multimodal network framework based on deep neural networks to extract depression features from multiple perspectives in different modalities and improve the accuracy of depression detection;

[0060] 4) Combine the audio feature extraction module, the text bidirectional long-short time memory network framework model and the video time distribution convolutional neural network model to detect the depressive state of the population by constructing a fusion network. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a flow chart of a method for auxiliary detection of depression in the population based on a multimodal deep neural network according to the present invention;

[0062] Figure 2 Schematic diagram of the audio modal feature extraction module proposed in the present invention;

[0063] Figure 3Schematic diagram of the text modality bidirectional long-short time memory network framework model proposed in the present invention;

[0064] Figure 4 Schematic diagram of the video modality time distribution convolutional neural network model proposed in the present invention;

[0065] Figure 5 It is a schematic diagram of the high-dimensional features of the fusion of three modalities proposed in the present invention. DETAILED DESCRIPTION

[0066] The present invention will be further described below with reference to the accompanying drawings, but the protection scope of the present invention is not limited thereto.

[0067] Everyday behaviors associated with depression include speech, text, and video actions. For audio, the pre-trained Wav2vec2.0 encoder is used to encode the audio sequence, and a feature extraction module (Transformer) is used to extract high-dimensional features. For text, the pre-trained Bert encoder is used to encode the text sequence, and a Bidirectional Long Short-Term Memory (BiLSTM) model is used to extract high-dimensional features. For video, since depression is not an immediate behavioral state and cannot be distinguished in a short period of time, a relatively long observation period is required to draw conclusions. Therefore, the visual modality needs to consider the dynamic characteristics of the data, namely temporal information. Simple LSTM or CNN methods have certain limitations due to the lack of temporal information and are not suitable for processing long sequences. Due to privacy concerns, sensors are used to collect action coordinates, including facial landmark coordinates, head movement coordinates, and eye gaze coordinates. After obtaining this data, a Temporally Distributed Convolutional Neural Network (T-CNN) model is used to extract high-dimensional features. Finally, the features of the three different modalities are fused through the network to output a binary detection result.

[0068] Figure 1 A flowchart of a method for assisting the detection of depressive states in a population based on a multimodal deep neural network is given. The present invention proposes a method for assisting the detection of depressive states in a population based on a multimodal deep neural network. Taking into account the certain differences in daily behavior between patients with depression and healthy people, high-dimensional features related to depression are extracted from the three modalities of audio, text, and video that are closely related to daily behavior. Considering that depression is not a temporary state and requires long-term observation to make a judgment, it is necessary to extract long-term dynamic features from the video modality, and use the T-CNN network to process sequence features containing the time dimension. Therefore, the present invention adopts a multimodal fusion method to extract features from different modalities, and then outputs binary labels after fusion through a self-attention module.

[0069] The specific operating steps of the present invention are:

[0070] Figure 2 A schematic diagram of the audio modal feature extraction module is given

[0071] 1) After reading the raw audio data, it is pre-processed and fed into the Wav2vec2.0 audio encoding module to extract audio features. The feature extraction module then extracts high-dimensional features of the encoded audio. The proposed Wav2vec2.0 audio encoding module uses a pre-trained Wav2vec2.0 model to extract audio clips.

[0072] The feature extraction module proposed in this invention is composed of a multi-head self-attention network, which is mainly composed of a multi-head self-attention mechanism structure and a feedforward neural network stack; the feedforward neural network includes two layers of fully connected layers, and for the input feature sequence Where T and d m The feature extraction module first projects x three times to obtain the query vector Q, key vector K and value vector V, and then divides the three into h parts to generate d h dimensional features, among which h is a positive number. The features of each part are passed through a multi-head self-attention mechanism structure and then output Q' to the next multi-head self-attention mechanism structure. At the same time, the input x is input into the feedforward neural network after passing through another multi-head self-attention mechanism structure, obtaining a weighted feature vector and mapping it to K' and V', which is then input into the next multi-head self-attention structure. The input of the next multi-head self-attention structure is Q', K', and V', and its output is sent to the last layer of the feedforward neural network to obtain the output features of each part. The outputs of h dimensions are concatenated to form the final output. The model mapping formula is as follows:

[0073]

[0074] K′,V′=max(0,Q′W1+b1)W2+b2

[0075]

[0076] Figure 3 A schematic diagram of the text modal bidirectional long short-term memory network framework model is given

[0077] 2) After the original audio is transcribed into text, data is sorted and garbled characters and symbols are cleaned up. After reading, it is input into the Bert text encoding module to encode the original text into a one-dimensional feature vector;

[0078] Non-numeric text data cannot be directly input into the model. It must be encoded and converted into numerical data before it can be used for model training and prediction. Traditional mainstream text encoding methods include the bag-of-words model and the word vector model. The bag-of-words model uses one-hot encoding, frequency encoding, or TF-IDF statistics to obtain word vector representations, and then represents the sentence as the sum of the word vectors. The bag-of-words model is simple to implement and has good interpretability. However, it cannot express the meaning of words or calculate semantic similarity (text similarity) between words and sentences, and the encoded vectors are very sparse. The word vector model, on the other hand, trains word vector representations based on the collocation relationship between words and context.

[0079] The BERT model also relies on the superposition of bidirectional Transformers. It is superior to previous methods because it is the first unsupervised, deeply bidirectional system for pre-training NLP. It uses deeper models and a massive corpus. The resulting embedding representation has a much higher accuracy than other methods when used for downstream tasks. In various downstream tasks, BERT can be selectively fine-tuned in a multi-task manner.

[0080] The embedding vector obtained after Bert encoding is sent to the bidirectional long short-term memory network to extract high-dimensional features. BiLSTM can better capture the bidirectional semantic dependency. The LSTM model is composed of the input word X at time t. t , cell state C t , temporary cell state Hidden state h t , forget gate f t , Memory Gate i t , output gate o t The calculation process can be summarized as follows: by forgetting the information in the cell state and updating the memory, the information useful for subsequent calculations is transmitted, while the useless information is discarded, and the hidden state h is output at each time step. t , where forgetting, memory and output are determined by the hidden state h at the previous moment t-1 and the current input X t The calculated forgetfulness f t , Memory Gate i t and output gate o t To control, we finally get the same hidden state sequence as the sentence. The formulas for each variable are as follows:

[0081] f t =σ(W f ·[h t-1 ,x t ]+b f ) (8)

[0082] it =σ(W i ·[h t-1 ,x t ]+b i ) (9)

[0083]

[0084]

[0085] o t =σ(W o ·[h t-1 ,x t ]+b o ) (12)

[0086] h t =o t ×tanh(C t ) (13)

[0087] The text modality network proposed in the present invention obtains a feature vector of (1, 768) after Bert encoding, and outputs a high-dimensional feature vector of (1, 256) after passing through the two-layer BiLSTM network. In order to avoid overfitting, a Dropout layer is added after each layer of the network in the BiLSTM module.

[0088] Figure 4 A schematic diagram of the video modality network is given

[0089] 3) The three-dimensional coordinate information of the subject's head movements during the interview is collected by wearing sensors, and the three-dimensional coordinate information of 68 key points on the subject's face during the interview is collected by video recording.

[0090] After reading the key point information, The feature vector of , where T is the time step and D is the feature dimension.

[0091] Taking the three-dimensional facial key point coordinate sequence and the head posture motion coordinate sequence as input, a time-distributed convolutional neural network is used to extract high-dimensional features; for the input facial coordinate point information and head posture motion information, two parallel T-CNN branches are used to learn features, and then the self-attention module is used to fuse the learned features and assign weights to different feature channels. The proposed T-CNN is a multi-layer structure consisting of five dilated convolution blocks and four maximum pooling layers. On the one hand, convolution blocks are used to explore important information within different perceptual ranges. On the other hand, the maximum pooling layer continuously reduces the feature resolution and gradually extracts important high-dimensional features. Therefore, T-CNN can be regarded as a feature learning module that extracts specific information about depression from multiple scales. Given an input The convolution operation of T-CNN can be expressed as:

[0092]

[0093] Where d is the dilation factor, k is the convolution kernel size, and b is the bias. Zero padding is used to keep the input and output shapes the same. In the network, the dilation factor is a multiple of 2 to obtain time information of different time spans on the path. The two parallel paths are connected using the ELU activation function.

[0094]

[0095] In order to keep the tensor size the same between modules, a 1×1 convolution layer is added to each dilated convolution block. Batch normalization is used at the end of the dilated convolution block to speed up the training process. Batch normalization can also alleviate the gradient vanishing problem. However, in order to preserve the distribution of different features, batch normalization is not used in the last dilated convolution block.

[0096] A maximum pooling layer is added after the first four dilated convolution blocks. The maximum pooling layer enables the output tensor to obtain the largest receptive field, which can gradually aggregate important information from long sequences. In addition, the maximum pooling layer can also reduce the sequence length, retain the most important part, and reduce the complexity of the model.

[0097] Each dilated convolution block takes a given input The output of the model is The output is the same size as the input. Therefore, in tasks with long input sequences, the output size may also be large. In order to expand the receptive field and deepen the network scale, the expansion coefficient will be too large, which will increase the computational cost and other side effects. Therefore, multiple different expansion factors d = (1, 2, 4) are added to the network. Secondly, in order to shorten the feature length, the maximum pooling layer naturally expands the receptive field of the adjacent dilated convolution blocks, reducing its input size to T / 2M (M = 2 n-1 , n=1, 2, 3, 4, 5), after giving the input original coordinate data, the output is a high-dimensional feature vector of (1, 256).

[0098] Figure 5 A schematic diagram of high-dimensional features that fuse three modalities is given;

[0099] 4) Three high-dimensional features are obtained from the three branch networks processing different modalities. The obtained features are concatenated in the first dimension to obtain a 1×768 feature vector. After input into the self-attention module and the fully connected layer, a set of binary labels representing the depressive state is obtained through classification.

[0100] The output after the self-attention module is (1, 768). Three fully connected (FC) layers are used in this paper: FC1 (512 neurons), FC2 (512 neurons), and FC3 (2 neurons). The first two FC layers are followed by a ReLU activation layer, and FC3 is followed by a softmax activation layer. This represents the depressive state of the input audio. The loss between the depressive state and its true value is estimated, denoted by L.

[0101] The above is a brief description of the present invention and is not limited to the above-mentioned working scope. As long as the ideas and working methods of the present invention are simply modified and applied to other equipment, or improvements and embellishments are made without changing the main conceptual principles of the present invention, they are all within the scope of protection of the present invention.

Claims

1. A method for auxiliary detection of depression in people based on multimodal deep neural network, characterized in that: The steps include: Step 1): Extract high-dimensional features of the depressive state using a feature extraction module based on the audio modality; The feature extraction module consists of a multi-head self-attention network, which is composed of a multi-head self-attention mechanism structure and a stacked feedforward neural network. The feedforward neural network includes two fully connected layers. The activation function of the first fully connected layer is ReLU, and the activation function of the second fully connected layer is a linear activation function. Step 2): Extract high-dimensional features of depression status using a bidirectional long-short-term memory network framework model based on text modality; The bidirectional long short-term memory network is a BiLSTM network composed of bidirectional LSTMs. The BiLSTM network uses a double-layer LSTM layer. Step 3): Extract high-dimensional features of depression state using a time-distributed convolutional neural network model based on the video modality; The temporal distributed convolutional neural network model consists of two parallel T-CNN branches. The T-CNN branch has a multi-layer structure, including five dilated convolution blocks and four maximum pooling layers. The dilated convolution blocks are used to obtain information within different perception ranges; the maximum pooling layers are used to reduce feature resolution and extract high-dimensional features. The specific steps include: 3.1) : Collect the 3D coordinate information of the subject's head movements during the interview by wearing sensors, and collect the 3D coordinate information of the subject's facial key points during the interview by recording video; The obtained head movement and three-dimensional coordinate information of key points are read and obtained The feature vector of , where T is the time step and D is the feature dimension; 3.2) : Using the 3D coordinate feature vector sequence of facial key points and the 3D coordinate feature vector sequence of head movements as input, a time-distributed convolutional neural network is used to extract high-dimensional visual features; Input the facial key point 3D coordinate feature vector sequence and the head movement 3D coordinate feature vector sequence to two parallel T-CNN branches for feature learning; enter , the convolution operation of the T-CNN branch is expressed as: （5） Where d is the dilation factor, k is the convolution kernel size, and b is the bias; zero padding is used to keep the input and output shapes of T-CNN the same; Step 4): Fusing the high-dimensional features of depression extracted from the three modalities, outputting binary labels through the self-attention module to detect depression in the population; the specific steps include: 4.1) Integrate the high-dimensional features extracted from the three branch models to obtain a vector I, which is input into the self-attention module. The self-attention module fuses the learned features and assigns weights to different feature channels. The resulting weight matrix represents the importance of different features of the three modalities. When the input vector I is input, the calculation process of the self-attention mechanism is: Calculate the three vector matrices of the fused vector I, namely matrix: （6） （7） （8） Calculate the attention score matrix : （9） （10） Calculate the output matrix : （11） in is a learnable parameter.

2. The method for auxiliary detection of depression in a population based on a multimodal deep neural network according to claim 1 is characterized in that: In the step 1), the specific steps include: Step 1.1): Audio sequence encoding Use the pre-trained Wav2vec2.0 model as the Wav2vec2.0 audio encoding module to extract audio clips; The audio clips of the subjects were recorded through video recording. After preprocessing with noise reduction, impurity removal, and mute removal, they were input into the Wav2vec2.0 audio encoding module, with the output feature dimension being 512 × the number of frames. The Wav2vec2.0 audio encoding module includes the vq-wav2vec quantization module and the Transformer structure. Step 1.2): Feature extraction module performs feature extraction The features output by the Wav2vec2.0 audio encoding module Input to the feature extraction module, where T and They are the length and dimension of the input respectively; the feature extraction module extracts features Perform three projections to obtain the query vector Q, key vector K and value vector V, and then split Q, K and V into parts, produce dimensional features, among which , Is a positive number, and the features of each part are output after a multi-head self-attention mechanism structure. ; At the same time, the input features After passing through another multi-head self-attention mechanism structure, it is input into the feedforward neural network to obtain a weighted feature vector and mapped to ; Input to the next multi-head self-attention structure, and send the output to the last layer of feedforward neural network to obtain the output features of each part. The outputs of the dimensions are connected in series to form the final output; the mapping formula of the feature extraction module is as follows: （2） （3） （4） in are learnable weights, For bias.

3. The method for auxiliary detection of depression in a population based on a multimodal deep neural network according to claim 1, characterized in that: In the step 2), the specific steps include: 2.1): Text sequence encoding The audio of the person being tested is transcribed to obtain text data, which is then cleaned and input into the Bert text encoding module. The Bert text encoding module is a pre-trained Bert model. The input is the transcribed text of each sample and the output is a one-dimensional feature vector of 1×768. 2.2): Bidirectional long short-term memory network extracts high-dimensional features of text The one-dimensional feature vector obtained in step 2.1) is input into the bidirectional long short-term memory network. The one-dimensional feature vector is first input into the front LSTM layer, and then the reverse time series is input into the back LSTM layer.

4. The method for auxiliary detection of depression in a population based on a multimodal deep neural network according to claim 1, characterized in that: In step 4), the presence or absence of depression is classified by learning the features of the three modalities. There are only two prediction results, and the predicted probabilities are and ; Use the cross entropy loss function, defined as follows: （1） In the formula: is the weighted cross entropy loss function, is the modal channel used, is the cross entropy loss function, is the multimodal feature vector of three-channel splicing, Is the relative The learned weights, Indicates the label of the sample, 1 for depression and 0 for no depression; Represents the probability that the sample is predicted to have depression.

5. The method for auxiliary detection of depression in a population based on a multimodal deep neural network according to claim 3 is characterized in that: In step 2.1), when the text is input to the Bert text encoding module, it is first encoded into a code that can be recognized by the model computer language; the text is first encoded into numbers according to the dictionary, which is a token embedding; when the input text is two sentences, it is separated by the [SEP] flag to obtain a segment embedding, where the element corresponding to the first sentence is encoded as 0 and the element corresponding to the second sentence is encoded as 1; when the input text element is position information, it is a position embedding; the three embeddings are combined and input to the Bert text encoding module.

6. The method for auxiliary detection of depression in a population based on a multimodal deep neural network according to claim 1, characterized in that: In step 3.2), the multiple of the expansion factor is 2, which is used to obtain time information of different time spans on the path, and the two parallel paths are connected by an activation function.

Images