Multi-mode depression detection method, system, medium and equipment

Abstract

The invention discloses a multi-modal depression detection method, system, medium and equipment, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining multi-modal data, including text, voice and vision; multi-modal data features are extracted; then performing information interaction processing on the multi-modal features through a cross-modal multi-head attention mechanism, splicing interaction features with the same target modals to obtain spliced interaction features, and introducing a multi-head self-attention mechanism to process the spliced interaction features to obtain processed spliced interaction features; a CubeMLP module is constructed based on the three MLP units, and the CubeMLP module mixes the splicing interaction features to obtain mixed multi-modal features; and flattening the mixed multi-modal features to obtain flattened mixed multi-modal features, sending the flattened mixed multi-modal features to a classifier to obtain a depression tendency result, and carrying out regression on the depression tendency by taking consistency correlation coefficient loss as a loss cost training model to obtain a depression detection result. And more efficient and more accurate depression identification can be realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal depression detection method, system, medium and device. Background Art

[0002] In recent years, people have paid more and more attention to the use of computers for auxiliary diagnosis of depression, and have achieved good results. Facial micro-expressions can convey more than half of people's information, so they are often used to classify depression. In the field of expression monitoring, networks such as facial action units and image classification dual streams have been introduced into the field of computer-assisted diagnosis of depression. At a time when the development of biometric recognition is becoming more and more popular, researchers have turned their attention to the field of speech. Speech is the most direct and convenient way for people to communicate with each other. It is non-invasive, but contains a large amount of emotional information that cannot be ignored. Generally speaking, people with depression have physiological and psychological states such as insomnia, excessive anxiety, and abnormal emotions, which will be directly connected in speech: people with depression often speak in a dull manner, repeat vocabulary, have a monotonous tone, and have a weak tone. Different studies have shown that unique signal information in speech can be associated with depression. For example, the Mutter team found through research that people with depression have narrow audio bands, more pauses, slow speech speed, and low pitch compared to speaking.

[0003] Nicholas' team found that non-depressed people have greater variability in speech space. For example, Jiang's team used a logistic ensemble regression network to extract the spectral and rhythmic features of speech to diagnose depression. For another example, Harati's team integrated speech detection of depression with neural networks for the first time. Improving the accuracy of depression recognition by combining multimodal data with deep learning algorithms has become a direction worthy of in-depth research.

[0004] However, existing methods still have some shortcomings and disadvantages. First, the correlation between sentences is weak during feature extraction, which may lead to the model's inability to capture contextual information and emotional changes. Second, the fusion of features from different modalities is relatively random, and there is a lack of systematic methods to effectively combine multimodal data, which may affect the recognition effect and robustness of the model. In summary, the existing methods lack analysis of the correlation between sentence features and feature fusion, which affects the accuracy and effectiveness of the final analysis results. Summary of the invention

[0005] In order to overcome the problems of the above-mentioned prior art in multimodal depression detection, such as weak correlation between sentences during feature extraction, arbitrary feature fusion of different modalities, and lack of verification of the generalization ability of the model on the data set, the main purpose of the present invention is to provide a multimodal depression detection method, system, medium and device.

[0006] To achieve the above object, the present invention adopts the following technical solution, a multimodal depression detection method, comprising the following steps:

[0007] A multimodal depression detection method, characterized in that it comprises the following steps:

[0008] Acquire multimodal data, including text data, voice data, and visual data;

[0009] Performing feature extraction on the multimodal data to obtain multimodal features;

[0010] Performing information interaction processing on the multimodal features through a cross-modal multi-head attention mechanism to obtain multimodal interaction features, splicing interaction features with the same target modality in the multimodal interaction features to obtain spliced ​​interaction features, and introducing a multi-head self-attention mechanism to enhance the spliced ​​interaction features to obtain enhanced spliced ​​interaction features;

[0011] A CubeMLP module is constructed based on three MLP units, and the CubeMLP module mixes the splicing interaction features to obtain mixed multimodal features;

[0012] The mixed multimodal features are flattened to obtain the flattened mixed multimodal features and sent to the classifier to obtain the depressive tendency results.

[0013] Also includes:

[0014] The splicing interaction features are processed by a multi-head self-attention mechanism to obtain processed splicing interaction features; the CubeMLP module mixes the processed splicing interaction features to obtain mixed multimodal features.

[0015] The MLP unit includes ReZero, a fully connected layer, an activation function and a normalization layer.

[0016] The feature extraction of the text data, voice data and visual data to obtain text features, voice features and visual features comprises the following steps:

[0017] Characterize the text data through BERT to obtain text features;

[0018] The speech data and visual data are characterized by BiGRU to obtain speech features and visual features.

[0019] A multimodal depression detection system, comprising:

[0020] A multimodal data processing module is used to obtain multimodal data, including text data, voice data and visual data; perform feature extraction on the multimodal data to obtain multimodal features;

[0021] The multimodal feature analysis module performs information interaction processing on the multimodal features through a cross-modal multi-head attention mechanism to obtain multimodal interaction features, splices the interaction features with the same target modality in the multimodal interaction features to obtain spliced ​​interaction features, introduces a multi-head self-attention mechanism to enhance the spliced ​​interaction features to obtain enhanced spliced ​​interaction features; constructs a CubeMLP module based on three MLP units, and the CubeMLP module mixes the spliced ​​interaction features to obtain mixed multimodal features;

[0022] The depression detection module is used to flatten the mixed multimodal features, obtain the flattened mixed multimodal features and send them into the classifier to obtain the depression tendency results.

[0023] Compared with the prior art, the beneficial effects of the present invention are:

[0024] The present invention constructs a new type of multi-head attention module. The model first obtains complementary information between modalities through a cross-modal multi-head attention mechanism, and then introduces a multi-head self-attention mechanism to obtain hidden information within the modality, so that the emotional information is fuller and richer, so as to solve the problem of incomplete feature representation of the existing model. By considering text, voice and visual data at the same time, the method can capture the patient's signs of depression more comprehensively. Data of different modalities can provide complementary information about the patient's emotional state, thereby reducing the misdiagnosis or missed diagnosis that may be caused by a single modality. Using a cross-modal multi-head attention mechanism to process multimodal features for information interaction can effectively fuse information between different modalities. This fusion helps to reveal the intrinsic connection between modalities and improve the ability to recognize depressive symptoms. By mixing the spliced ​​interactive features through the CubeMLP module, the representation ability of the features can be further enhanced, so that the classifier can judge the depressive tendency based on richer and deeper features. Flattening the mixed multimodal features and feeding them into the classifier helps to improve the performance of the classifier, thereby more accurately identifying the patient's depressive tendency. The multi-head self-attention mechanism can further improve the model's ability to capture key features, allowing the model to pay more attention to information related to depression detection. The MLP unit includes ReZero, fully connected layer, activation function and return; BERT is used to extract features from text data, and BiGRU is used to extract features from speech and visual data. These advanced models can effectively extract meaningful features from their respective modalities, thereby improving detection accuracy.

[0025] In summary, the beneficial effects of this technical solution include but are not limited to improving the accuracy of depression detection, reducing the possibility of misdiagnosis and missed diagnosis, enhancing the model's ability to fuse information from different modalities, and improving the model's training efficiency and generalization ability through modular design. These effects have important practical application value for clinical diagnosis and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0027] Figure 1 It is a schematic diagram of the structure of the model of the present invention;

[0028] Figure 2 BERT structure diagram of the present invention;

[0029] Figure 3 It is a schematic diagram of the GRU structure of the present invention;

[0030] Figure 4 It is a schematic diagram of the BiGRU structure of the present invention;

[0031] Figure 5 Schematic diagram of the attention mechanism structure of the present invention;

[0032] Figure 6 It is a schematic diagram of the Multi-Head Self-Attention structure of the present invention;

[0033] Figure 7 It is a schematic diagram of the ReZero structure of the present invention;

[0034] Figure 8 It is a detailed schematic diagram of the MLP of the present invention;

[0035] Fig. 9 It is a schematic diagram of the process structure of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0037] The model framework proposed in this application is as follows Figure 1 As shown in the figure, a multimodal depression detection method framework consists of four parts, namely multimodal feature extraction, attention mechanism, improved CubeMLP and prediction head. The model takes three modal data as input: text (T), speech (A) and vision (V). The specific contents are as follows:

[0038] (1) Feature extraction: For three different modal data, BERT is used to extract text modal features, and BiGRU is used to extract audio and video features. (2) Attention mechanism: By constructing Cross Attention, six information interaction features are obtained: text to speech, vision, speech to text, vision, and vision to text, speech. Then, the information interaction features with the same target modality are concatenated to fuse the semantically relevant information of the other two modalities and obtain a single modal feature vector containing rich emotional semantic information, thereby achieving the effect of information enhancement for the target modality. Self Attention is performed after Cross Attention so that the model can better capture the relationship between elements in the sequence.

[0039] (3) Improved CubeMLP: The semantic correlation between the three information-enhanced unimodal features is captured by using the attention mechanism to obtain the final fused multimodal features. The improved CubeMLP block consists of three MLP units, each of which is designed to mix multimodal features on its own axis. Each MLP consists of ReZero, a fully connected layer, an activation function, and a normalization layer.

[0040] (4) Prediction head: Flatten the mixed multimodal features and then feed the flattened features into the classifier to predict the emotional tendency or depression level of depression.

[0041] The BERT model is a pre-trained language representation model proposed by Google in 2018. It is based on the Transformer architecture and has learned rich language knowledge and contextual understanding capabilities through pre-training on a large-scale text corpus. The characteristics of the BERT model include a bidirectional attention mechanism that can simultaneously capture contextual information on the left and right sides of the input sequence. The training process of BERT can be mainly divided into two stages: pre-training and fine-tuning. In the pre-training stage, the BERT model learns language knowledge through two main pre-training tasks, while using a masked language model to predict the next sentence. Then, the model is fine-tuned using task-specific training data to adapt to the new task. The BERT model can adapt to different NLP tasks through pre-training and fine-tuning. The pre-training stage enables the model to learn rich language knowledge and contextual understanding capabilities, while the fine-tuning stage adapts the model to specific task requirements. The BERT model structure is as follows Figure 2 shown.

[0042] Where E=[E Dynamic ,E changes …,E County ] is the input text; E c =[E Dynamic ,E changes …,ECounty ] is the word vector; E S =[E A , E B …E Z ] is the segment vector; E N =[E 1 ,E 2 …E n ] is the position vector; T=[T 1 ,T 2 ,…,T n ] is the output vector matrix; Trm is the 12-layer Transformer encoder.

[0043] As an improved model of RNN, the GRU model effectively solves the gradient vanishing and gradient exploding problems in RNN. Compared with the LSTM model, it has a simpler structure and fewer parameters, which can improve efficiency. The basic units of the GRU network are as follows: Figure 3 , where the update gate is responsible for filtering and storing information; the reset gate is responsible for selectively forgetting the previous node state.

[0044] The parameter update calculation method of the GRU network is as shown in formula (1-4).

[0045] Z t =σ(W Z ×[h t-1 ,x t ]) (1)

[0046] r t =σ(W r ×[h t-1 ,x t ]) (2)

[0047] h' t =tanh(W h' ×[r t ×h t-1 ,x t ]) (3)

[0048] h t =(1-z t )×h t-1 +z t ×h' t (4)

[0049] Where: W represents the weight matrix; x t Indicates the input information of the current node; h t-1 Represents the hidden state of the previous node; r t Indicates resetting the door state; z t Indicates the update gate state; h' trepresents the candidate hidden state; h t represents the hidden state passed to the next node, t is the tth moment; σ represents the Sigmoid activation function; tanh is the tanh activation function; × represents the Hadamard product; Represents a vector concatenation operation; Multiply vector elements.

[0050] In order to fully understand the contextual information in the text, a bidirectional GRU model is introduced. The BiGRU model consists of a forward GRU and a backward GRU. By splicing the feature information in two directions, the bidirectional gated recurrent unit (BiGRU) is a neural network that combines a bidirectional recurrent neural network (Bi-RNN) and a gated recurrent unit (GRU). Its main idea is to consider both forward and backward information at each time step to better capture contextual information. The gating mechanism of GRU allows BiGRU to selectively forget some historical information and selectively obtain new information from the current input, which helps to reduce the gradient vanishing problem and accelerate the training process.

[0051] The structure of BiGRU is as follows Figure 4 As shown, it consists of two independent GRUs, namely GRU f and GRU b , where GRU f Represents the forward GRU network; GRU b represents the backward GRU network; g M n =[g M n,1 ,g M n,2 ,…,g M n,K ] represents the output feature sequence of the multimodal dataset; h M n =[h M n,1 ,h M n,2 ,…,h M n,K ] represents the feature sequence in multimodal dataset processing; M represents the modality type, and M∈{A,V}; Y A n represents the short-term speech emotion characteristics of the nth utterance; Y V n Represents the nth short-term expression emotion feature. The output feature sequence g of the multimodal dataset M n As the input of BiGRU, it is input to GRU in sequence f and GRU bto be processed.

[0052] GRU f From the first feature g of the sequence M n,1 Start by gradually processing each feature vector in the speech and expression feature sequence, while passing information from the previous feature vector to the next feature vector. b From the last feature g of the sequence M n,K Start by gradually processing each feature vector in the speech and expression feature sequence, while passing information from the subsequent feature vectors to the previous feature vectors. GRU f The output and GRU b The output of is continued to be spliced ​​at each time step to obtain the output feature of the BiGRU module, that is, the short-term speech emotion feature Y of the nth utterance A n and short-term expression emotion characteristics Y V n The parameter update calculation method of the BiGRU network is as shown in formula (5-6). Represents a vector concatenation operation.

[0053]

[0054] Cross attention and Self attention are two commonly used attention mechanisms in natural language processing. The main difference between the two is that the objects for calculating the correlation and weighted sum are different. Self attention is used for a single input sequence to capture the relationship between elements in the sequence; while Cross attention acts on multiple different input sequences to establish correlations between different sequences. The attention mechanism structure in the model of this application is as follows: Figure 5 .

[0055] Among them, Y A ,Y T ,Y V Respectively represent the feature vectors of sound, text, and video after feature extraction; Y V-A Represents the interactive feature information from mode V to mode A; Y T-A Represents the interactive feature information from mode T to mode A; Y A-T Represents the interactive feature information from mode A to mode T; Y V-T Represents the interactive feature information from mode V to mode T; Y T-V Represents the interactive feature information from mode T to mode V; Y A-V Represents the interactive feature information from mode A to mode V; Respectively represent the sound, text, and video feature vectors enhanced by the cross-modal multi-head attention mechanism; Respectively represent the sound, text, and video feature vectors enhanced by the two attention mechanisms.

[0056] In this application, Cross attention uses a cross-modal multi-head attention mechanism. Specifically, each cross-modal mechanism repeatedly strengthens the target modality with low-level features from another source modality by learning the attention between the two modal features, so that one modality can receive information from another modality. Taking audio-vision as an example, the audio information (A) is passed to the visual information (V), represented by AV. In order to capture the emotional information interaction between any two modalities, first input any two unimodal features Y in the model α and Y β The concatenation can obtain the multimodal feature D, which serves as the input of the key matrix (K) and the value matrix (V) in the multi-head attention mechanism as the source modality. In addition, the unimodal feature Y α and Y β They are respectively used as the target modal features and as the input of the query matrix (Q). The calculation process of the multimodal feature D is expressed as:

[0057]

[0058] The information interaction feature process from mode β to mode α using the cross-modal information interaction mechanism is recorded as Y β-α , we need to transform the unimodal feature vector Y α Project it with the multimodal feature D and use the scaled dot product formula in the multi-head attention mechanism to calculate the single-modal Y α The specific calculation process of attention in D is as follows:

[0059]

[0060] Where i represents a specific head in the multi-head attention mechanism, and W D1 and W D2 ,W Q is the mapping matrix, is the scaling factor.

[0061] The target modality features assigned with attention weights to each head are concatenated and transformed linearly to obtain the information interaction features from modality b to modality a:

[0062]

[0063] Where h represents the number of heads in the multi-head attention mechanism, W o is the weight parameter.

[0064] In this process, each target modality uses a cross-modal attention mechanism to continuously update its sequence of external information (source modality) to learn features that are meaningfully associated with the cross-modality. The features from the modality are transformed into a set of different key-value pairs to interact with the target modality, where the relevant cross-modal features can be captured by the cross-modal attention mechanism to obtain the final information interaction features. Using the above cross-modal information interaction mechanism, six information interaction features can be generated, which can be represented as Y T-A , Y T-V , Y A-T , Y A-V , Y V-T , Y V-A .

[0065] Then, the information interaction features with the same target modality features are spliced ​​together, aiming to integrate the effective emotional information of other modalities to enhance the feature representation of the target modality and obtain a single-modal feature vector containing rich emotional semantic information.

[0066]

[0067] Where W 1 , W 2 , W 3 , W 4 , W 5 , W 6 is the weight parameter matrix in the fully connected layer, b 1 , b 2 , b 3 , b 4 , b 5 , b 6 are bias values, which are all trainable parameters in the model, and GELU represents the activation function.

[0068] After Cross Attention, the model introduces a multi-head self-attention mechanism to process the semantically enhanced feature vector, so that the model can more effectively obtain the hidden information in the modality and represent the features more completely. Figure 6 .

[0069] Q=[q 1 q 2 ... q n ]=W Q ×[x 1 x 2 ... x n ] (16)

[0070] K=[k 1 k2 ... k n ]=W k ×[x 1 x 2 ... x n ] (17)

[0071] V=[v 1 v 2 ... v n ]=W V ×[x 1 x 2 ... x n ] (18)

[0072]

[0073] M(Q,K,V)=C(A 1 ,A 2 ,...,A n )W 0 (20)

[0074] Where: W Q , W K , W V and W 0 is the weight matrix; Q, K and V are query vector, key vector and value vector respectively; q=[q 1 ,q 2 ,…,q n ],k=[k 1 ,k 2 ,…,k n ],v=[v 1 ,v 2 ,…,v n ] are the components of each query vector, key vector and value vector respectively; A n is the output of each self-attention layer; is the attention weight, is the Softmax function, is the dot product scaling factor; M is the combined output of the multi-head self-attention layer; C is the Concat function; is the feature vector enhanced by the two attention mechanisms.

[0075] Researchers from the University of California, San Diego (UCSD) proposed a neural network structure improvement method called ReZero, which can dynamically accelerate the propagation of high-quality gradients and arbitrary deep signals. ReZero mainly brings the following two benefits: (1) Ability to train deeper neural networks: Learning signals can be effectively transmitted in deep neural networks, which enables the model to train some networks that were previously untrainable. As the depth increases, the advantages of the ReZero architecture become more obvious. (2) Faster convergence speed: Compared with conventional residual networks with normalization operations, ReZero converges significantly faster. When ReZero is applied to Transformer, its convergence speed is 56% faster than that of the general Transformer on the enwiki8 language modeling benchmark, reaching 1.2BPB. When ReZero is applied to ResNet, it can achieve 32% acceleration and 85% accuracy on CIFAR 10.

[0076] ReZero initializes all network layers to the same mapping. In each layer, the researchers introduced a residual connection about the input signal x and a trainable parameter α for adjusting the output F(x) of the current network layer, namely:

[0077] x i+1 =x i +α i F(x i ) (twenty one)

[0078] Among them, x is the input signal; L represents the number of network layers; α is a trainable parameter used to adjust the output F(x) of the current network layer.

[0079] At the beginning of training, α is set to zero. This makes the gradients corresponding to all the parameters that make up the transformation F disappear at the beginning of neural network training. After that, these parameters dynamically generate appropriate values ​​to optimize training during the training process. The improved network structure is as follows Figure 7 As shown:

[0080] Among them, L represents the number of layers of the network; x is the input signal; α is a trainable parameter used to adjust the output F(x) of the current network layer.

[0081] After ReZero is introduced into the model, the activation function is adjusted to GELU to maintain the stability of the model. m ∈R L×D Expanded to R in the second dimension L×1×D , and then concatenate them along the expansion axis into multimodal features Y∈R L×M×D, where M is the number of modes. The multimodal features are then passed to N layers of stacked CubeMLP for mixing. The CubeMLP module consists of three MLP units, each of which is designed to mix multimodal features on its respective coordinate axis. Specifically, the first MLPf L : R L×*×* →R L′×*×* It is designed to perform sequential mixing acting on the L axis. The second modal MLPf M : R *×M×* →R *×M′×* is a mixed mode MLP acting on the M axis. The third MLPf D : R *×*×D →R *×*×D′ Channel mixing is performed on the D axis. Each MLP unit consists of two fully connected layers and an activation function GELU. The fully connected layer can also be regarded as two affine transformations. The MLP details are as follows: Figure 8 .

[0082] Among them, GELU is the activation function; ReZero is a method for improving the neural network structure.

[0083] Consider the first sequential hybrid MLP unit on the L axis. The tensor Y can be viewed as Y *，m，d ∈R L×1×1 Y is a set of vectors, where (m,d)∈{(1,1),(1,2),..,(2,1),(2,2),(M,D)}. *，m，d is a vector of the mth modality and the dth channel. Each affine transformation in the sequence mixture MLP can be expressed as:

[0084] Aff L (Y *,m,d )=W L Y *,m,d +B L (twenty two)

[0085] Where W L ∈R L×L′ and B L ∈R L′ are learnable parameters represented by two matrices. L' is the dimension reduction on the L axis and is a hyperparameter. Formula (22) shows that all Y *，m，d Shared parameter W L and B L . Therefore, the complete MLP unit can be mathematically represented as:

[0086]

[0087] Where: σ is nonlinear activation, LN is layer normalization. The output tensor U∈R of the first MLP unit isL′×M×D It can be seen as U *，m，d ∈R L′×1×1 The set of vectors where (m,d)∈{(1,1),(1,2),…,(2,1),(2,2),…,(M,D)}.

[0088] Like the first MLP unit on the L axis, the output V∈R of the second MLP unit on the M axis L′×M′×D and the output Y'∈R of the third MLP unit on the D axis L′×M′×D′ can be regarded as their elementary vector V l，*，d ∈R 1×M′×1 and Y' l，m，* ∈R1×1×D', where M' and D' are the reduced dimensions on the M-axis and D-axis respectively. As in formula (23), V l,*,d and Y' l,m,* They can be calculated as formula (24) and formula (25) respectively. L'×M'×D' It is a mixed multi-modal feature.

[0089]

[0090] After obtaining the mixed multimodal feature Y′∈R L'×M'×D' Afterwards, the model flattens it into Y′∈R L'M'D' Then the flattened features are fed into the classifier f c : R L'M'D' →R, predicting emotional tendency or depression level.

[0091] In the training process of multimodal sentiment analysis, this application uses the mean absolute error (MAE) as the loss function, which is a regression task:

[0092]

[0093] Where N is the number of samples; y i is the true value; is the predicted value; L mae is the MAE loss function. MAE is used to calculate the absolute error between the prediction and the true value. MAE has a significant effect on small errors than high-order errors, allowing the model to achieve better accuracy on subtle emotions. Therefore, MAE is often used as a key performance indicator for sentiment analysis.

[0094] For the emotional tendency of depression obtained by depression detection, this application uses the consistency correlation coefficient (CCC) loss as the loss cost to train the model to regress the depression tendency:

[0095]

[0096] in, is the standard deviation of the measured values; is the standard deviation of the true value; is the mean of the measured values; is the mean of the true values; is the covariance between the test value and the true value; L CCC is the CCC loss function.

[0097] CCC is widely used to estimate depression

[28] , because it is not only invariant to scale and position changes, but also includes measures of relevance and accuracy

[29] .

[0098] Example

[0099] For the verification of the dataset, the CMU-MOSI dataset collected the comments and opinions of 89 different narrators on YouTube on topics such as movies. There are 93 videos in total, which are divided into 2199 video clips, and each video clip has a corresponding emotion label. The range of the emotion label of each video clip is [-3,3], -3 represents strong negativity, and +3 represents strong positivity. Since most researchers use binary classification for emotion classification, samples with labels greater than or equal to 0 are marked as positive, that is, positive emotions; samples with labels less than 0 are marked as negative, that is, negative emotions. The dataset is further divided, 1151 video clips in 52 videos are used as training sets, 296 video clips in 10 videos are used as validation sets, and 752 video clips in 31 videos are used as test sets.

[0100] CMU-MOSEI upgrades CMU-MOSI by expanding the size of the dataset, with more utterances, samples, narrators, and topics, with the same labeling style as CMU-MOSI. The dataset contains 5,000 videos, divided into 23,453 video clips, 1,000 different narrators, and 250 different topics. The dataset is further divided into 16,216 video clips from 2,250 videos as training sets, 1,835 video clips from 300 videos as validation sets, and 4,625 video clips from 679 videos as test sets.

[0101] The AVEC2019DDS dataset is obtained from audio-visual recordings of clinical interviews with patients. The interviews are conducted by virtual agents to exclude human interference. Unlike the above two datasets, each modality in AVEC2019 provides several different features. For example, the acoustic modality consists of deep features extracted by MFCC, eGeMaps, and VGG. In previous studies, Hao et al. found that MFCC and AU-pose are the two most discriminative features in acoustic and visual modalities, respectively. Therefore, for the purpose of simplicity and efficiency, this application only uses MFCC and AU-poses features for depression detection. The dataset is annotated by PHQ-8 scores in the interval [0,24], and a larger PHQ-8 score means a more severe depressive tendency. There are 163 training samples, 56 validation samples, and 56 test samples in this benchmark dataset.

[0102] The basic information of the three datasets is shown in Table 1:

[0103] Table 1 Statistics of MOSI, MOSEI, and AVEC2019 datasets

[0104] Statistics CMU-MOSI CMU-MOSEI AVEC2019 Train 1284 16326 163 Val 229 1871 56 Test 686 4659 56 Total 2199 22856 273

[0105] Following previous work, the performance of the model is evaluated in the form of classification and regression. The classification results are discretized from the regression results and reflect the accuracy of the basic sentiment prediction, i.e., negative (negative) or positive (positive). Classification metrics include binary classification accuracy (Acc-2) and weighted F1 score (F1). Previous work reported in two categories, such as negative / non-negative (including 0) class and negative / positive class (excluding 0). This application reports the results of these two metrics using split tags, where the scores on the left are negative / non-negative and the scores on the right are negative / positive classes. In addition, a more fine-grained 7-classification accuracy (Acc-7) is also reported. Regression metrics include the mean absolute error (MAE), which directly calculates the error between the prediction and the real label. Pearson correlation score (Pearson correlation), which reflects the correlation with the true value of the label annotated by humans. For all the above metrics, higher values ​​indicate better model performance.

[0106]

[0107] The neural network architecture of this application is developed and designed based on the Pytorch framework. A large number of scientific experiments have been conducted to verify the feasibility and effectiveness of the methods proposed in this section. The following table shows the environment configuration of the software and hardware required during the experiment. The configuration of the CMU-MOSI and CMU-MOSEI experimental servers is shown in Table 2.

[0108] Table 2 Experimental environment based on CMU-MOSI and CMU-MOSEI

[0109] Name CMU-MOSI CMU-MOSEI CPU AMD Ryzen 7 7745HX with Radeon Graphics AMD EPYC 9754 128-Core Processor GPU NVIDIA GeForce RTX 4060 Laptop GPU NVIDIA Corporation RTX 3090 GPU Operating system Windows 11 (x64) Ubuntu (22.04) Python environment Python 3.10 Python 3.10

[0110] For multimodal feature extraction, L is set to 100 for sentiment analysis and 1000 for depression. Due to the different lengths of the samples, this application sets the sequences with smaller lengths to zero and truncates the sequences with larger lengths to match the length. For each modal feature, this application sets D to 128. In the study of this application, M is fixed to 3 because the model has 3 participating modalities (t, a and v). For L, M and D'. In the experiment, this application found that the improved CubeMLP is a very effective structure, and when N is set to 3, the performance can reach the state-of-the-art level. During the training process, the learning rate is initialized to 0.004 and multiplied by 0.1 every 50 epochs. After obtaining the best hyperparameter combination, we save and compress the model as the best_model_val.pth file, and then use this model for depression detection.

[0111] Table 3 Key parameters of different CMU-MOSI and CMU-MOSEI model experiments

[0112] name CMU-MOSI CMU-MOSEI Text modal vector dimension 768 768 Audio modal vector dimension 74 74 Video modal vector dimension 47 35 learning rate 0.004 0.004 Batch Size 128 64 Optm Adam Adam

[0113] For multimodal sentiment analysis, Table 4 lists the comparative experimental results of each model on the CMU-MOSI and CMU-MOSEI multimodal datasets. The experimental results show that the improved CubeMLP model that integrates the multi-head attention mechanism has achieved good results on the specified performance indicators compared with other baseline models, and the generalization is very good. In the CMU-MOSI dataset, compared with the original CubeMLP model, MAE reached 0.765, a decrease of 0.005; Corr reached 0.764, a decrease of 0.003; Acc-2 reached 86.1, an increase of 0.5; F1-Score reached 86.1, an increase of 0.6; Acc-7 reached 45.6, an increase of 0.1. In the CMU-MOSEI dataset, compared with the original CubeMLP model, Acc-2 reached 85.2, an increase of 0.1; F1-Score reached 84.8, an increase of 0.3.

[0114] Table 4 Experimental results of different models of CMU-MOSI and CMU-MOSEI on two datasets

[0115]

[0116] For multimodal depression detection, the results of the models are shown in Table 5. For other methods, Baseline uses late fusion and averages the final predictions of the involved modalities. EF introduces simple language features and word length features to estimate the depression level. Bert-CNN&Gated-CNN uses a gate mechanism to fuse information obtained from related modalities. Multi-scaleTemporal Dilated-CNN uses an expanded convolutional neural network to extract multimodal features using a larger receptive field.

[0117] Experimental results show that the improved CubeMLP model that integrates the multi-head attention mechanism proposed in this application has improvements in all evaluation indicators. Through the cross-modal multi-head attention mechanism, the information between multiple modalities is interactively supplemented, and the complementary information between modalities is effectively obtained. Then, the multi-head self-attention mechanism is introduced to more effectively obtain the hidden information within the modality. On the AVEC2019 DDS dataset, the model of this application achieved a CCC of 0.592 and a MAE of 4.25, achieving better results in both CCC and MAE. Compared with the original CubeMLP, the CCC increased by 0.011 and the MAE decreased by 0.14, both achieving the best results.

[0118] Table 5 Experimental results of different models on the AVEC2019 dataset

[0119]

[0120] Example 2

[0121] In order to explore the impact of different multimodal combinations on the depression detection effect of the model, a multimodal sentiment analysis task was performed on the CMU-MOSI dataset, and the experimental results are shown in Table 6. The results show that for unimodal sentiment analysis, among the combinations of text modality (T), voice modality (A) and visual modality (V), the text-related combination performs better, indicating that the text modality contains the richest emotional information. When the three modalities (T+A+V) of text (T), voice (A) and vision (V) are used at the same time, the model performs best, which verifies the advantages of multimodal sentiment analysis over unimodal sentiment analysis, and can help the model to mine more useful emotional information, thereby improving the accuracy of sentiment analysis. In other words, the model of this application uses a cross-modal multi-head attention mechanism to interactively supplement the information between multiple modalities, and uses multi-head self-attention to perform semantic enhancement on the internal modality, which can effectively improve the depression detection effect of the model.

[0122] Table 6 Experiments on the impact of different module structures on CMU-MOSI

[0123] No. T A V MAE Corr Acc-2 F1-Score Acc-7 1 √ 0.850 0.748 80.8 80.9 39.3 2 √ 0.840 0.731 80.5 80.6 39.4 3 √ 0.903 0.722 81.9 82.0 39.3 4 √ √ 0.804 0.753 81.7 81.8 42.6 5 √ √ 0.802 0.7 80.7 80.8 39.7 6 √ √ 0.864 0.721 82.7 82.7 41.8 7 √ √ √ 0.765 0.764 86.1 86.1 45.6

[0124] In order to evaluate the effectiveness of the attention mechanism module, the improved multimodal feature extraction module and the improved CubeMLP module in the model, an ablation experiment was conducted on the CMU-MOSEI dataset, and the experimental results are shown in Table 7. Among them, MLP means that neither the attention mechanism module nor the improved feature extraction module and the improved CubeMLP module are used. MLP+BiGRU means that the model only uses the improved multimodal feature extraction module. Similarly, MLP+BiGRU+ReZero means that the model uses the improved multimodal feature fusion module and the improved CubeMLP module. MLP+BiGRU+Self attention+Cross-attention+ReZero means that the attention mechanism module, the improved multimodal feature extraction module and the improved CubeMLP module are integrated at the same time. The experimental results show that the improved CubeMLP model that integrates the multi-head attention mechanism has the largest accuracy improvement and the best effect. Compared with the model without multi-head attention mechanism, the fully integrated model has a 0.006 reduction in MAE, a 0.027 reduction in Corr, a 0.029 increase in Acc-2, a 0.027 increase in F1-Score, and a 0.101 increase in ACC-7. The lack of any functional module will lead to a decrease in the performance indicators of the model, which effectively confirms the importance of each module in the improved CubeMLP model.

[0125] Table 7 Ablation experiments on the MOSI dataset

[0126]

[0127] Based on the CubeMLP model, an improved model is proposed that uses the multi-head attention mechanism to fully obtain the complementary and hidden information between modalities. The activation function is adjusted on the original MLP module, and ReZero is introduced to improve the training efficiency and model stability. Experiments have shown that the indicators of the improved CubeMLP model evaluation model proposed in this application have been improved both on the multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI, and on the depression estimation dataset AVEC2019.

[0128] In order to verify the effectiveness and generalization of the model, this application conducted in-depth experiments on two different types of datasets, multimodal sentiment analysis and multimodal depression detection datasets. The results show that the method of this application is competitive with the current most advanced methods in sentiment analysis, the model has good generalization performance, and has made great progress in depression detection.

[0129] It should be noted that, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0130] The above embodiments are merely examples of the present invention and do not limit the protection scope of the present invention. All designs that are the same or similar to the present invention fall within the protection scope of the present invention.

Claims

1. A multimodal depression detection method, characterized in that: The following steps are involved: Acquire multimodal data of clinical patients; Performing feature extraction on the multimodal data to obtain multimodal features; Performing information interaction processing on the multimodal features through a cross-modal multi-head attention mechanism to obtain multimodal interaction features, splicing interaction features with the same target modality in the multimodal interaction features to obtain spliced ​​interaction features, and introducing a multi-head self-attention mechanism to enhance the spliced ​​interaction features to obtain enhanced spliced ​​interaction features; A CubeMLP module is constructed based on three MLP units, and the CubeMLP module mixes the enhanced splicing interaction features to obtain mixed multimodal features; The mixed multimodal features are flattened to obtain the flattened mixed multimodal features and sent to the classifier to obtain the depressive tendency results.

2. A multimodal depression detection method as claimed in claim 1, characterized in that: The MLP unit includes ReZero, a fully connected layer, an activation function and a normalization layer.

3. A multimodal depression detection method as claimed in claim 1, characterized in that: The multimodal features include text data, voice data and visual data; The step of extracting features from the multimodal data to obtain multimodal features comprises the following steps: Characterize the text data through BERT to obtain text features; The speech data and visual data are characterized by BiGRU to obtain speech features and visual features.

4. A multimodal depression detection method as claimed in claim 3, characterized in that: The obtained processed splicing interaction features, The following steps are involved: Calculate the attention weight for each attention head based on the query and key, and use the attention weight to perform weighted summation to obtain the weighted feature representation of each modality; The weighted features output by each attention head are concatenated to obtain the fused multimodal interaction features; The interaction features with the same target modality that each attention head assigns the attention weight to are concatenated, and after a linear transformation, the concatenated interaction features are obtained; A multi-head self-attention mechanism is introduced to process the splicing interaction features to obtain processed splicing interaction features.

5. A multimodal depression detection system, characterized in that: include: A multimodal data processing module, used to obtain multimodal data, including text data, voice data and visual data; Performing feature extraction on the multimodal data to obtain multimodal features; A multimodal feature analysis module performs information interaction processing on the multimodal features through a cross-modal multi-head attention mechanism to obtain multimodal interaction features, and splices the interaction features with the same target modality to obtain spliced ​​interaction features; A CubeMLP module is constructed based on three MLP units, and the CubeMLP module mixes the splicing interaction features to obtain mixed multimodal features; The depression detection module is used to flatten the mixed multimodal features, obtain the flattened mixed multimodal features and send them into the classifier to obtain the depression tendency results.

6. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

7. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

Images