Multi-mode multi-factor depression identification system fusing emotion information

Abstract

The invention provides a multi-modal multi-factor depression recognition system fusing emotion information, relates to the technical field of multi-modal fusion, and provides a transfer learning and multi-dataset training method to extract more accurate emotion features so as to solve the problem of insufficient emotion recognition of the existing method. According to the method, feature extraction is carried out by taking clauses as units, and complete context information is reserved, so that language and emotion fluctuation of the user can be better understood. In order to optimize multi-modal data fusion, the invention adopts a fusion method based on an attention mechanism, which not only can more effectively capture the dependency relationship among the modals, but also can fully extract key information in each modality, thereby improving the overall detection effect. Meanwhile, the influence of gender difference and external knowledge is also considered, the accurate recognition and understanding ability of the model on the emotions of the users with different genders is improved by introducing the gender difference and an external common knowledge base, and the adaptability and accuracy of the model are further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal fusion, and in particular to a multimodal and multi-factor depression recognition system integrating emotion information. Background Art

[0002] Depression has become the second largest health killer in the world, seriously affecting the quality of life and health level of human beings. Traditional methods of diagnosing depression mainly rely on the clinical judgment of doctors, through self-reports of patients, psychological assessment scales (such as the Beck Depression Inventory) and clinical observations of doctors. However, this method not only requires high medical costs, but also has a long diagnosis cycle. At the same time, the symptoms of depression are complex and have individual differences, which can easily lead to misdiagnosis or missed diagnosis. Therefore, there is an urgent need for more efficient, accurate and low-cost early detection methods for depression. With the development of science and technology, the use of computers to assist depression detection has become a hot research direction in academia in recent years. Among them, depression detection technology based on multimodal data (text and voice) is particularly eye-catching. By analyzing the patient's voice and text data, computers can identify potential depressive symptoms, thereby providing doctors with important auxiliary diagnostic tools.

[0003] Currently, computer-assisted depression detection methods mainly include three methods: text-based, speech-based, and multimodal fusion-based.

[0004] Text-based depression detection, especially deep learning methods, can identify depression more accurately than traditional methods by automatically extracting emotional features from text. Deep learning models such as BERT and LSTM can handle more complex language features and understand emotional changes in context. Unlike traditional machine learning methods, deep learning does not require manual feature extraction, but learns how to identify depression-related patterns in text through large amounts of data. BERT learns the deep semantic relationships of language on a large corpus through pre-training, while LSTM can capture long-term dependencies in text, which is particularly suitable for processing emotional fluctuations and complex contexts. For example, Wang et al. analyzed Weibo data and combined it with a deep learning model to achieve results that are superior to traditional machine learning methods.

[0005] Speech-based depression detection, especially deep learning methods, uses automatic feature extraction technology to identify depression by analyzing deep emotional information in speech. Deep learning models, such as convolutional neural networks (CNNs) and long short-term memory networks (LSTMs), can extract complex emotional features from raw speech signals or spectrograms without manually designing features. CNNs can process local features of speech and are suitable for identifying short-term changes in sound, while LSTMs can capture long-term dependencies in speech and identify long-term emotional fluctuations such as rhythm and speech rate changes in the speech of patients with depression. For example, He et al. combined manual features with deep features and used deep convolutional neural networks for classification prediction, achieving good results.

[0006] Depression detection based on multimodal data has become a hot topic of research by combining two or more data sources to identify depression. Compared with a single modality, multimodal data is more robust. When one modality is missing, it can still be predicted through other modalities. At the same time, it provides richer information to help identify depression more accurately. At present, depression detection methods based on multimodality are mainly divided into three categories: feature-level fusion, decision-level fusion, and model fusion.

[0007] Feature-level fusion stitches together features from multiple modalities before classification. For example, Gong et al. extracted text, speech, and video features, stitched them together, and used random forests for classification prediction. Decision-level fusion combines the results after independent predictions from each modality, such as weighted summation or voting. Meng et al. weighted fused the prediction results of speech and video, showing that multimodality performs better than single modality. Model fusion improves prediction accuracy by deeply fusing features from different modalities and capturing the relationship between modalities. Li et al. constructed a multimodal hierarchical attention model.

[0008] Although multimodal depression detection technology has made some research progress, existing methods still face many challenges in practical applications: (1) Insufficient feature extraction: Most current multimodal depression detection methods rely on feature extraction based on phonemes, sentences, or words. Although this method is effective in some scenarios, it fails to capture long-distance dependencies and contextual information in user language, resulting in incomplete context understanding. Especially in multi-round conversations, emotional fluctuations in the context and changes in user emotions are often difficult to capture, which may lead to bias and misjudgment of emotion recognition.

[0009] (2) Limitations of emotion feature extraction: Most existing emotion recognition technologies rely on preset emotion dictionaries. These emotion dictionaries cover a large number of emotional words, but they usually have certain limitations. For example, these dictionaries often lack the ability to capture subtle changes in emotions and cannot effectively reflect the diversity and complexity of user emotions. In particular, for atypical emotional expressions (such as the emotional expressions of patients with latent depression), existing emotion dictionaries may not be able to fully recognize them. In addition, cultural differences, language differences, and individual differences in emotion dictionaries may also affect the universality and accuracy of emotion feature extraction.

[0010] (3) The uniqueness of multimodal feature fusion: Multimodal data (text and voice) each contains unique information. Text data mainly reflects the user's language content and expression, while voice data reveals the user's emotional fluctuations. In order to make full use of these two modal data, an efficient fusion method must be adopted. However, many existing multimodal fusion methods are still relatively simple, usually using rough fusion methods such as splicing or weighted averaging, resulting in the inability to fully extract and integrate key information in different modal data. For example, the emotional color in the text information may not be fully integrated with the emotional information of the voice, thus affecting the overall emotion recognition effect.

[0011] (4) Ignoring external knowledge and individual differences: Many current depression recognition systems often fail to take into account external factors related to user feelings, psychological state, life background, etc. For example, factors such as gender, age, and social background often play a crucial role in the manifestation of depression. Ignoring these factors will cause the model to have a biased understanding of the user's emotions. In addition, the symptoms of patients with depression usually vary from person to person, and the traditional "one-size-fits-all" approach is difficult to capture these individual differences, resulting in a decrease in the universality and accuracy of the model. Summary of the invention

[0012] In view of the shortcomings of the prior art, the purpose of the present invention is to propose a multimodal and multifactor depression recognition system that integrates emotional information, including a preprocessing module, a text feature extraction module, a speech feature extraction module, a multimodal fusion module and a multifactor fusion module; The preprocessing module is used to preprocess the original speech of the user to be detected, and obtain the preprocessed text as a clause and a speech waveform graph, wherein the abscissa of the speech waveform graph is time, and the ordinate of the speech waveform graph is amplitude; The text feature extraction module is used to add [CLS] at the start position of the preprocessed clause and [SEP] at the end position, segment the clause after adding the marker through a marker to obtain segmented clauses; extract features from the segmented clauses through BERT to obtain semantic features. f ts; The segmented clauses are processed through the text sentiment extraction model to obtain the text sentiment features f te , the semantic features f ts and the text sentiment features f te Integrate into text features f t ; The text emotion extraction model is obtained in the following way: Obtain a clause in the SST-2 data set as a first input sample, and use the emotion label corresponding to the clause in the SST-2 data set as a first output sample, wherein the first input sample and the first output sample constitute a first training sample, and perform model training on the RoBERTa model based on multiple first training samples until the RoBERTa model converges to obtain a trained RoBERTa model, retain the parameters and encoder structure of the trained RoBERTa model, and replace the output layer of the trained RoBERTa model with a fully connected layer FC to obtain a text emotion extraction model; The speech feature extraction module is used to extract features from the speech waveform to obtain deep speech features. f ad , input the original speech into the speech emotion extraction model to obtain the speech emotion features f ae , the deep features of speech f ad and voice emotion features f ae Concatenate to get complete speech features f a ; The speech emotion extraction model is obtained by training the Wav2Vec2.0 model based on multiple second training samples, wherein the second training samples include a second input sample and a second output sample, the second input sample is a speech sample in the IEMOCAP data set, and the second output sample is a speech emotion feature corresponding to the speech sample in the IEMOCAP data set; The multimodal fusion module is used to f t and voice features f a , generate text modality features S t , intra-modal features of speech S a , features in speech associated with text S ta and features in text that are associated with speechS at , according to the text modality features S t , intra-modal features of speech S a , features in speech associated with text S ta and features in text that are associated with speech S at , calculate the first fusion multimodal feature and the second fusion multimodal features ; The multi-factor fusion module is used to determine the gender of the user to be detected. When the user to be detected is male, the first fused multimodal feature is classified by a male-oriented classification model. Processing is performed to obtain the depression prediction result of the user to be detected. When the user to be detected is a female, the second fusion multimodal feature is classified by the female-oriented classification model. Processing is performed to obtain the depression prediction result of the user to be detected.

[0013] Optionally, the preprocessing module is specifically used to obtain the original speech of the user to be detected, determine the triplet <start time, end time, clause> of the user to be detected in the original speech, use regular expressions to perform data cleaning on the clause, and obtain the preprocessed clause; perform pre-emphasis processing on the original speech to obtain pre-emphasized speech, specifically, enhance the high-frequency part of the original speech, improve the high-frequency resolution of the original speech, perform frame processing on the pre-emphasized speech, apply a Hanning window function to the framed speech to enhance the speech signal, obtain an enhanced speech signal, and draw a speech waveform diagram based on the enhanced speech signal.

[0014] Optionally, the text feature extraction module extracts features from the segmented clauses using BERT to obtain semantic features. f ts ,include: Get word embedding, position embedding and tag embedding in the segmented clauses, add the word embedding, position embedding and tag embedding to get the added embedding, input the added embedding into the Transformer encoder for encoding to get the semantic features f ts .

[0015] Optionally, the speech feature extraction module extracts features from the speech waveform to obtain speech deep features. f ad ,include: Grayscale the speech waveform to obtain a grayscale waveform, normalize and standardize the grayscale waveform to obtain a processed waveform, wherein the pixel values ​​in the processed waveform are within the range of [0, 1], and the pixel values ​​have a uniform mean and standard deviation, and the processed waveform is converted into a one-dimensional matrix and input into TCN to obtain deep features of speech. f ad .

[0016] Optionally, the multimodal fusion module is specifically used to transform text features f t Input into the self-attention Transformer to obtain the text modality features S t , which is specifically achieved through the following formula: ; ; ; ; in, W Qt , W Kt , W Vt is the parameter matrix, T represents the transpose of a matrix, d k is the vector dimension, sum It means sum; The voice features f a Input into the self-attention Transformer to obtain the features within the speech modality S a , which is specifically achieved through the following formula: ; ; ; ; in, W Qa , W Ka , W Va is the parameter matrix; Through the cross attention mechanism, according to the features in the text modality S t and intra-modal features of speech S a, calculate the features in speech associated with text S ta , expressed by the following formula: ; Through the cross attention mechanism, according to the features in the text modality S t and intra-modal features of speech S a , calculate the features associated with speech in the text S at , expressed by the following formula: ; Features in text modality S t , intra-modal features of speech S a , features in speech associated with text S ta Features associated with speech S at Perform concatenation operation to obtain matrix U M =[ S t , S a , S ta , S at ], the matrix U M =[ S t , S a , S ta , S at ], input two layers Transformer Encoder In the above example, we obtain the first fusion multimodal feature , which is specifically expressed by the following formula: =Transformer ( U M ); Features in text modality S t , intra-modal features of speech S a , features in speech associated with text S ta Features associated with speech S at Perform the sum operation and get the matrix U F=[ S t , S a , S ta , S at ], the matrix U F =[ S t , S a , S ta , S at ], input two layers Transformer Encoder In the second fusion multimodal feature , which is specifically expressed by the following formula: =Transformer ( U F ).

[0017] Optional, Cross-Attention ( S t , S a ) is calculated by the following formula: ; ; ; ; Similarly, Cross-Attention ( S a , S t ) is calculated by the following formula: ; ; ; .

[0018] Optionally, the multi-factor fusion module uses a male-oriented classification model to fusion the first multi-modal feature Processing is performed to obtain the depression prediction results of the user to be tested, including: Acquire inferences about emotions and feelings xReact , based on the common sense knowledge base COMET, the preprocessed clauses and reasoning relations xReact Processing is performed to obtain common sense knowledge, which is specifically achieved through the following formula: ; in, s ti The first clause in the preprocessed i words, s ki For the i Common sense knowledge corresponding to each word, ⊕ indicates concatenation; Common sense knowledge s ki , input into BERT to encode and get common sense knowledge s ki The embedding representation of b i , which is specifically achieved through the following formula: ; The embedding representation of all common sense knowledge forms the sentence vector B = { b 1 , b 2 ,…, b m}, input the sentence vector B Transformer , get the target common sense knowledge K u , achieved through the following formula: ; Target common sense knowledge K u And the first fusion multimodal features Splice and get the first feature , the first feature enter BiGRU In the classifier, we get features containing forward and backward information , which is specifically expressed by the following formula: ; The features that will contain forward and backward information Input the fully connected layer to obtain the depression prediction result of the user to be tested y’ , which is specifically expressed by the following formula: .

[0019] Optionally, the multi-factor fusion module uses a female-oriented classification model to fusion the second multi-modal feature Processing is performed to obtain the depression prediction results of the user to be tested, including: Acquire inferences about emotions and feelings xReact , based on the common sense knowledge base COMET, the preprocessed clauses and reasoning relationsxReact Processing is performed to obtain common sense knowledge, which is specifically achieved through the following formula: ; in, s ti The first clause in the preprocessed i words, s ki For the i Common sense knowledge corresponding to each word; Common sense knowledge s ki , input into BERT to encode and get common sense knowledge s ki The embedding representation of b i , which is specifically achieved through the following formula: ; The embedding representation of all common sense knowledge forms the sentence vector B = { b 1 , b 2 ,…, b m}, input the sentence vector B Transformer , get the target common sense knowledge K u , achieved through the following formula: ; Target common sense knowledge K u and the second fusion multimodal features Splice and get the second feature , the second feature Input into the MLP classifier to obtain the depression prediction result of the user to be tested y’ , which is specifically expressed by the following formula: .

[0020] Optional, The specific calculation is achieved through the following formula: ; ; ; in, , and is the parameter matrix, , and is the bias parameter, is the output of the first hidden layer, is the output of the second hidden layer.

[0021] Optionally, the multimodal and multifactor depression recognition system integrating emotional information further includes a loss module; The loss module is used to use the cross entropy loss function as the loss function, which is specifically expressed by the following formula: ; in, For the j The actual results of depression in samples For the j Depression prediction results of samples; The value of the loss function is used to update the parameters in the preprocessing module, text feature extraction module, speech feature extraction module, multimodal fusion module and multi-factor fusion module.

[0022] The beneficial effects of adopting the above technical solution are: The present invention proposes to extract more accurate emotion features through transfer learning and multi-dataset training methods, thereby solving the problem of insufficient emotion recognition in existing methods. In addition, the present invention also retains complete contextual information by performing feature extraction on a clause basis to better understand the user's language and emotional fluctuations. In order to optimize multimodal data fusion, the present invention adopts a fusion method based on an attention mechanism, which can not only more effectively capture the dependencies between modalities, but also fully extract key information from each modality, thereby improving the overall detection effect. At the same time, the present invention also takes into account the influence of gender differences and external knowledge, and introduces gender differences and an external common sense knowledge base to improve the model's ability to accurately identify and understand the emotions of users of different genders, thereby further improving the model's adaptability and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the structure of a multi-modal and multi-factor depression recognition system integrating emotional information in an embodiment of the present invention; Figure 2 It is a schematic diagram of the flow of a multi-modal and multi-factor depression recognition system integrating emotional information in an embodiment of the present invention; Figure 3 The figure is a schematic diagram of a process of determining a triplet of a user to be detected in an original speech in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0025] In view of the problems existing in the prior art, the present invention provides a multi-modal and multi-factor depression recognition system integrating emotional information. Figure 1 , including a preprocessing module, a text feature extraction module, a speech feature extraction module, a multimodal fusion module and a multi-factor fusion module; Among them, in the text feature extraction module, BERT is first used to extract text semantic features to obtain sentence vectors containing semantic information, and then the pre-trained RoBERTa model is used to extract the emotional information contained in the text, and then the semantic features and emotional features are spliced ​​to form text features; in the speech feature extraction module, TCN is first used to extract deep speech features from the speech waveform, and then the pre-trained Wav2Vec2.0 is used to extract speech emotional features from the original speech signal, and the deep speech features and speech emotional features are spliced ​​to form speech features; in the multimodal fusion module, text features and speech features are deeply fused based on the self-attention mechanism and the cross-attention mechanism to obtain deeply fused modal features; the multi-factor fusion module will build two classification models for depression recognition based on the different characteristics of men and women, and introduce external knowledge, use the common sense knowledge base COMET for common sense reasoning, and obtain user emotional changes. Finally, the fused features are sent to different classifiers according to the differences between men and women to obtain the user's detection label, that is, the depression prediction result.

[0026] The purpose of the present invention is to improve the accuracy and robustness of the automatic depression detection system, especially in terms of emotion feature extraction, context understanding, multimodal data fusion, gender differences and external knowledge integration. In order to improve the deficiencies of the prior art, the present invention proposes to extract more accurate emotion features through transfer learning and multi-dataset training methods, thereby solving the problem of insufficient emotion recognition in existing methods. In addition, the invention also extracts features in units of clauses to retain complete contextual information to better understand the user's language and emotional fluctuations. In order to optimize multimodal data fusion, the present invention adopts a fusion method based on an attention mechanism, which can not only more effectively capture the dependencies between modalities, but also fully extract key information from each modality, thereby improving the overall detection effect. At the same time, the invention also takes into account the influence of gender differences and external knowledge, and improves the model's ability to accurately recognize and understand the emotions of users of different genders by introducing gender differences and an external common sense knowledge base, further improving the adaptability and accuracy of the model.

[0027] Combination Figure 2 , the specific contents of each module include: The preprocessing module is used to preprocess the original speech of the user to be detected, and obtain the preprocessed text as a clause and a speech waveform graph, wherein the abscissa of the speech waveform graph is time, and the ordinate of the speech waveform graph is amplitude; The preprocessing module is specifically used to obtain the original speech of the user to be detected, and determine the triple of the user to be detected in the original speech <start time, end time, clause>; Specific combination Figure 3 , for the original speech signal, first identify the current speaker, that is, determine whether the current speaker is the user to be detected, that is, Figure 3 For the participants in , when the current speaker is a robot, determine whether there is a next sentence; when the current speaker is the user to be detected, update the <start time, end time, clause> triple, adjust the start time and end time according to the identity of the speaker of the adjacent sentence, if the speakers of the adjacent sentences are all users to be detected, that is, participants, only update the clause, and then determine whether there is a next sentence. If the speaker of the adjacent sentence is not the user to be detected, it is necessary to update the triple start time or end time, and then determine whether there is a next sentence; if there is a next sentence, return: identify the current speaker, if there is no next sentence returned, then end, thereby obtaining the triple <start time, end time, clause> of the user to be detected.

[0028] Use regular expressions to clean the clauses to remove irrelevant characters (such as " <an> ”" <fam> ”" <sync>”, etc.), to avoid interference, and obtain the preprocessed clause; pre-emphasize the original speech to obtain the pre-emphasized speech, specifically, enhance the high-frequency part of the original speech to remove the influence of lip radiation and improve the high-frequency resolution of the original speech. Since the speech signal is a non-stationary signal, but has short-term stability in a short time, the pre-emphasized speech is framed, the frame length is set to 25ms, the frame shift is 10ms, and the overlapping segmentation method is used for transition. The Hanning window function is applied to the framed speech to enhance the speech signal to obtain the enhanced speech signal, and a speech waveform is drawn according to the enhanced speech signal, wherein the abscissa of the speech waveform is time, and the ordinate is amplitude.

[0029] When testing the system of the present invention, the DAIC-WOZ dataset is used. The DAIC-WOZ dataset has the following characteristics: although there are pauses when the participants answer questions, their utterances are coherent, related to each other, and complementary to each other. Different from the previous research that extracts features based on a single utterance only, the present invention regards these continuous utterances as a complete clause to fully preserve the user context.

[0030] The text feature extraction module is used to add [CLS] at the start position of the preprocessed clause and [SEP] at the end position, segment the clause after adding the marker through a marker to obtain segmented clauses; extract features from the segmented clauses through BERT to obtain semantic features. f ts ; The segmented clauses are processed through the text sentiment extraction model to obtain the text sentiment features f te , the semantic features f ts and the text sentiment features f te Integrate into text features f t ; The text emotion extraction model is obtained in the following way: A clause in the SST-2 dataset is obtained as the first input sample, and the emotion label corresponding to the clause in the SST-2 dataset is taken as the first output sample. The first input sample and the first output sample constitute a first training sample. Based on multiple first training samples, a RoBERTa model is trained until the RoBERTa model converges to obtain a trained RoBERTa model. The parameters and encoder structure of the trained RoBERTa model are retained, and the output layer of the trained RoBERTa model is replaced with a fully connected layer FC to obtain a text emotion extraction model. After obtaining the text emotion extraction model, it is migrated to the segmented clauses to obtain the emotion features of the clauses. That is to say, the final text emotion extraction model can be used to extract text emotion features from clauses.

[0031] Among them, BERT is used to extract features from the segmented clauses to obtain semantic features. f ts ,include: Get the token embeddings, position embeddings and segment embeddings in the segmented clauses, add the token embeddings, position embeddings and segment embeddings together to get the added embeddings, and input the added embeddings into the Transformer encoder for encoding to get the semantic features. f ts .

[0032] The speech feature extraction module is used to extract the features of the speech waveform through TCN to obtain the deep features of the speech. f ad , input the original speech into the speech emotion extraction model to obtain the speech emotion features f ae , the deep features of speech f ad and voice emotion features f ae Concatenate to get complete speech features f a ; The speech emotion extraction model is obtained by training the Wav2Vec2.0 model based on multiple second training samples, wherein the second training samples include a second input sample and a second output sample, the second input sample is a speech sample in the IEMOCAP data set, and the second output sample is a speech emotion feature corresponding to the speech sample in the IEMOCAP data set; The training process specifically includes: The historical speech is obtained from the IEMOCAP dataset, and the speech matrix is ​​constructed based on the historical speech, and the speech matrix is ​​input into the FeatureEncoder module. After convolution and activation processing, the dense representation of the speech is obtained. Then, the global information is captured by the Transformer module in the Context Network, and the mask operation is performed and input into the Quantization Module to convert the continuous features into discrete features. Finally, the processed features are sent to the linear layer to obtain the predicted speech emotion features. Based on the predicted speech emotion features and the speech emotion features corresponding to the historical speech in the IEMOCAP dataset, the parameters in the Wav2Vec2.0 model are updated until the model converges to obtain the speech emotion extraction model. The best model parameters are saved and migrated to the original speech to obtain the emotion features of the speech. That is to say, the trained speech emotion extraction model can be used to obtain the emotion features of the speech from the original speech.

[0033] Since the waveform contains information such as the energy, pitch, loudness and timbre of the speech, it is more robust than the spectrogram. Since the waveform is related to time, the traditional CNN is not suitable for processing time series data, while TCN can effectively process images and extract time series features. Therefore, the present invention uses TCN for feature extraction.

[0034] Therefore, the speech waveform is subjected to feature extraction to obtain the deep features of the speech. f ad ,include: The speech waveform is grayed out, and the grayed waveform is obtained by weighted average simplification calculation. The grayed waveform is normalized and standardized to obtain a processed waveform. The pixel values ​​in the processed waveform are within the range of [0, 1], and the pixel values ​​have a uniform mean and standard deviation. The processed waveform is converted into a one-dimensional matrix and input into TCN to obtain deep speech features. f ad Specifically, to ensure that the input dimensions of each layer are consistent, the padding operation is performed first, and then the convolution calculation is performed through the dilated causal convolution layer. After the convolution, pruning, weight normalization, activation, dropout and residual connection are performed, and finally the deep features of the speech are output through the linear layer. f ad .

[0035] Multimodal fusion methods in automatic depression detection are mainly divided into three categories: feature-level fusion, decision-level fusion, and model fusion. The first two are shallow fusions. Although they are effective, they cannot fully capture the relationship between the modality and the modality, so a deep fusion method is needed. The present invention proposes a model fusion method based on self-attention and cross-attention, which uses self-attention to capture the information within the modality, and cross-attention to strengthen the important correlation features between the modalities, which is specifically realized through a multimodal fusion module.

[0036] The multimodal fusion module is used to f t and voice features f a , generate text modality features S t , intra-modal features of speech S a , features in speech associated with text S ta and features in text that are associated with speech S at , according to the text modality features S t , intra-modal features of speech S a , features in speech associated with text S ta and features in text that are associated with speech S at , calculate the first fusion multimodal feature and the second fusion multimodal features ; The multimodal fusion module is specifically used to combine text features f t Input into the self-attention Transformer to obtain the text modality features S t , can be expressed by the following formula: Transformer ( ); Specifically, the corresponding query vector (Q), key vector (K), value vector (V) and weight matrix are calculated, and the weight matrix is ​​normalized. The weight matrix is ​​then multiplied by V to obtain the self-attention output, and then the self-attention output is summed to obtain the text modal features containing important information in the modality. S t , which is specifically achieved through the following formula: ; ; ; ; in, W Qt , W Kt , W Vt is the parameter matrix, T represents the transpose of a matrix, d k is the vector dimension, sum It means sum; The voice features f a Input into the self-attention Transformer to obtain the features within the speech modality S a , can be expressed by the following formula: = Transformer ( ); Specifically, the corresponding query vector (Q), key vector (K), value vector (V) and weight matrix are calculated, and the weight matrix is ​​normalized. The weight matrix is ​​then multiplied by V to obtain the self-attention output, and then the self-attention output is summed to obtain the speech modality features containing important information in the modality. S a , which is specifically achieved through the following formula: ; ; ; ; in, W Qa , W Ka , W Va is the parameter matrix; Secondly, considering the different effects of speech on text and text on speech, this paper uses two cross-attention mechanisms to find the dependency between text and speech. Specifically, through the cross-attention mechanism, according to the features within the text modality S t and intra-modal features of speech S a , calculate the features in speech associated with text S ta , expressed by the following formula: ; Specifically, Cross-Attention ( S t , S a ) includes: according to the text features f t Calculate , according to the voice characteristics f a Calculate , , then the attention matrix is ​​normalized and compared with Multiply them to get the cross attention output, and then sum them to get the features associated with the text in the speech , which is calculated by the following formula: ; ; ; ; Through the cross attention mechanism, according to the features in the text modality S t and intra-modal features of speech S a , calculate the features associated with speech in the text S at , expressed by the following formula: ; in, Cross-Attention ( S a , S t ) includes: according to the speech features f a Calculate , according to the text features f t Calculate , Get the attention matrix, normalize it, and then Multiply them to get the cross-attention output, and sum the cross-attention output to get the features associated with speech in the text , which is calculated by the following formula: ; ; ; .

[0037] Existing studies have shown that gender is closely related to depression, and women are at a higher risk of depression than men. At the same time, in order to gain a deeper understanding of the feelings of users of different genders and perceive changes in user emotions, the present invention builds a multi-factor depression detection model that integrates gender information and common sense knowledge.

[0038] The multi-factor depression detection model is designed based on the different characteristics of male and female users. The male classification model (M model) and the female classification model (F model) are designed respectively. The main differences between the male and female models are reflected in two aspects: the first is that when obtaining the matrix U in the multimodal fusion module, the M model concatenates the summed self-attention output and cross-attention output, while the F model uses the summation operation, so when obtaining the text modality intra-feature S t , intra-modal features of speech S a , features in speech associated with text S ta Features associated with speech S at Afterwards, the M model performs the concatenation operation and the F model performs the summation operation.

[0039] Specifically, for the features within the text modality S t , intra-modal features of speech S a , features in speech associated with text S ta Features associated with speech S at Perform concatenation operation to obtain matrix U M =[ S t , S a , S ta , S at ], the matrix U M =[ S t , S a , S ta , S at ], input two layers Transformer Encoder In the above example, we obtain the first fusion multimodal feature ,Right now Figure 1 The deep fusion in is specifically expressed by the following formula: =Transformer ( U M ); Features in text modality S t , intra-modal features of speech S a , features in speech associated with text S ta Features associated with speech S at Perform the sum operation and get the matrix U F =[ S t , S a , S ta , S at ], the matrix U F =[ S t , S a , S ta , S at ], input two layers Transformer Encoder In the second fusion multimodal feature ,Right now Figure 1 The deep fusion in is specifically expressed by the following formula: =Transformer ( U F ).

[0040] The second main difference between the male and female models is in the classifier part. The M model uses the BiGRU classifier, while the F model uses the MLP classifier. The specific content is implemented through the multi-factor fusion module.

[0041] The multi-factor fusion module is used to determine the gender of the user to be detected. When the user to be detected is male, the first fused multimodal feature is classified by a male-oriented classification model. Processing is performed to obtain the depression prediction result of the user to be detected. When the user to be detected is a female, the second fusion multimodal feature is classified by the female-oriented classification model. Processing is performed to obtain the depression prediction result of the user to be detected.

[0042] In the multi-factor fusion module, an external common sense knowledge base COMET is introduced for common sense reasoning, giving the machine reasoning ability and understanding the causal relationship in the discourse. Among them, the common sense knowledge base COMET can be reasoned from the perspectives of user reaction (Reaction), intention (Intent), etc., and provide additional information. The present invention selects the reasoning relationship related to the user's emotions and feelings xReact , extract relevant knowledge. COMET is a Seq2Seq structure, which is divided into two parts: decoder and encoder. The encoder is based on BERT to generate corresponding feature vectors, and the decoder is based on GPT to generate discrete common sense knowledge related to clauses and reasoning relations.

[0043] Among them, the first fusion multimodal feature is Processing is performed to obtain the depression prediction results of the user to be tested, including: Acquire inferences about emotions and feelings xReact , based on the common sense knowledge base COMET, the preprocessed clauses and reasoning relations xReact Processing is performed to obtain common sense knowledge. Specifically, in the COMET encoder, the clauses and reasoning relationships during preprocessing are first xReact After concatenation, masking, deletion, filling and other operations are performed to generate feature vectors. After encoding, the feature vectors are sent to the decoder for operations such as pooling and linear transformation to generate clauses and reasoning relationships. xReact The relevant common sense knowledge is implemented through the following formula: ; in, s ti The first clause in the preprocessed i words, s ki For the i Common sense knowledge corresponding to each word, ⊕ indicates concatenation; Common sense knowledge s ki , input into BERT to encode and get common sense knowledge s ki The embedding representation of b i , which is specifically achieved through the following formula: ; The embedding representation of all common sense knowledge forms the sentence vector B = { b 1 , b 2 ,…, b m }, input the sentence vector B Transformer , get the target common sense knowledge K u , achieved through the following formula: ; Target common sense knowledge K u And the first fusion multimodal features Splice and get the first feature , the first feature enter BiGRU In the classifier, we get features containing forward and backward information , which is specifically expressed by the following formula: ; The features that will contain forward and backward information Input the fully connected layer to obtain the depression prediction result of the user to be tested y’ , which is specifically expressed by the following formula: .

[0044] Among them, the second fusion multimodal feature is Processing is performed to obtain the depression prediction results of the user to be tested, including: Acquire inferences about emotions and feelings xReact , based on the common sense knowledge base COMET, the preprocessed clauses and reasoning relations xReact Processing is performed to obtain common sense knowledge. Specifically, in the COMET encoder, the clauses and reasoning relationships during preprocessing are first xReact After concatenation, masking, deletion, filling and other operations are performed to generate feature vectors. After encoding, the feature vectors are sent to the decoder for operations such as pooling and linear transformation to generate clauses and reasoning relationships. xReact The relevant common sense knowledge is implemented through the following formula: ; in, s ti The first clause in the preprocessed i words, s ki For the i Common sense knowledge corresponding to each word; Common sense knowledge s ki , input into BERT to encode and get common sense knowledge s ki The embedding representation of b i , which is specifically achieved through the following formula: ; The embedding representation of all common sense knowledge forms the sentence vector B = { b 1 , b 2 ,…, b m }, input the sentence vector B Transformer , get the target common sense knowledge K u , achieved through the following formula: ; Target common sense knowledge K u and the second fusion multimodal features Splice and get the second feature , the second feature Input into the MLP classifier to obtain the depression prediction result of the user to be tested y’ , which is specifically expressed by the following formula: .

[0045] in, The specific calculation is achieved through the following formula: ; ; ; in, , and is the parameter matrix, , and is the bias parameter, is the output of the first hidden layer, is the output of the second hidden layer.

[0046] Among them, a multimodal and multifactorial depression recognition system integrating emotional information also includes a loss module; The loss module is used to use the cross entropy loss function as the loss function, which is specifically expressed by the following formula: ; in, For the j The actual results of depression in samples For the j Depression prediction results of samples; The value of the loss function is used to update the parameters in the preprocessing module, text feature extraction module, speech feature extraction module, multimodal fusion module and multi-factor fusion module.

[0047] Thus, the present invention has effectively improved the accuracy and adaptability of depression detection by integrating modal features with common sense knowledge features. The specific steps are: First, the algorithm takes the user's text data, voice data, and depression labels as input, and the goal is to output a set of depression recognition prediction results for all participants. In the data reading phase, each user's text data, voice data, and corresponding labels are read in turn. Next, the number of iterations and batch size required for model training are set to ensure the parameter settings of the training process.

[0048] Subsequently, the text feature extraction module and the voice feature extraction module are used to extract text features and voice features from the user's text and voice data respectively, and the two are fused to obtain multimodal features.

[0049] Then, we use the common sense knowledge base to obtain external knowledge of the text, enhance the relevant background in multiple rounds of dialogue, and further understand the emotions in the dialogue. At the same time, we also consider the impact of gender factors and subdivide the model into two models: male and female.

[0050] Finally, depression prediction is performed through a specific recognition formula to obtain the user's recognition result.

[0051] During the training process, the model uses the set number of iterations and batch size to process the data in batches, uses the loss function to calculate the model error, and updates the model parameters through the gradient descent optimization algorithm (Adam). Finally, the algorithm outputs a set of depression recognition prediction results for all users, completing the multimodal depression recognition task.

[0052] The present invention solves the deficiencies of existing methods in feature extraction when performing depression detection. First, the present invention obtains the environment of multi-round dialogue by constructing clauses. At the same time, BERT is used for text feature extraction. This model can handle long-distance dependencies well through the Transformer architecture and can capture context changes in multi-round dialogues. In terms of speech feature extraction, the present invention uses TCN to extract deep features from speech signals. Therefore, the present invention effectively captures long-distance dependencies and contextual information, especially the capture of emotional fluctuations and changes in multi-round dialogues.

[0053] In terms of emotion feature extraction, the present invention uses RoBERTa and Wav2Vec2.0 to extract emotion features in text and speech. Compared with the traditional method that relies on emotion dictionaries, this method can more flexibly identify the user's emotional changes, especially for delicate emotional expressions and atypical emotions (such as the emotions of patients with latent depression).

[0054] In addition, in terms of multimodal feature fusion, the present invention uses self-attention mechanism and cross-attention mechanism, which can deeply explore the relationship between text and speech, and effectively solve the problem that information cannot be fully integrated when feature fusion is performed by simple splicing or weighted averaging in existing methods. Through these mechanisms, the emotional features in text and speech can complement each other, thereby improving the accuracy and robustness of emotion recognition.

[0055] Finally, in the multi-factor fusion module, the present invention takes gender differences into account and introduces common sense reasoning (such as the COMET knowledge base), further enhancing the system's emotional understanding ability by reasoning about user emotional changes. The introduction of gender differences and external knowledge bases enables the present invention to provide personalized emotional analysis based on different user characteristics, especially in the identification of depression, which can better adapt to the emotional expressions of different users, thereby improving the universality and accuracy of the model.

[0056] Based on the above scheme, the present invention carried out the following experiments: The present invention uses two types of data sets, the first is an emotion data set for pre-training the model, and the second is a depression data set for verifying the effectiveness of the model. Emotion data sets are divided into text and voice emotion data sets. The text emotion data set used in the present invention is SST-2, which comes from Stanford University and contains movie reviews, which conform to daily language habits. SST-2 provides two types of emotion labels for each sentence: positive (1) and negative (0). The voice emotion data set is the IEMOCAP data set provided by the SAIL Laboratory of the University of Southern California, which contains 151 dialogues, 7433 sentences, and a duration of 12 hours. The emotion labels are divided into six categories: neutral, happy, sad, angry, excited, and depressed. Due to data imbalance, the present invention performs four classification tasks of angry, happy, neutral, and sad, and uses 90% of the data for training and 10% for testing.

[0057] To verify the effectiveness of the automatic depression recognition model, this paper uses the public dataset DAIC-WOZ. This dataset is based on the conversation between a robot (Ellie) and the participants, and the robot is remotely controlled. DAIC-WOZ was provided by the AVEC workshop. The data includes video features, audio, transcribed text, depression labels, gender, and PHQ-8 scores. There are 189 participants in total, divided into 107 training sets, 35 validation sets, and 47 test sets.

[0058] At the same time, in order to evaluate the performance of the model, the present invention uses the common evaluation indicators in the field of depression identification - precision, recall and F1 value to verify the effectiveness of the model.

[0059] First, in order to verify the effectiveness of the multimodal fusion part, the present invention removes the multi-factor fusion part and only focuses on the experimental results of the multimodal depression recognition model SCAED. Next, Table 1 shows the results of the comparative experiment and the ablation experiment.

[0060] Table 1 Comparative experiment and ablation experiment results model Precision Recall F1 GSM 74.0% 67.6% 67.0% BERT 68.8% 78.6% 72.7% BiLSTM(t) 80.0% 80.9% 80.4% ResNet34 57.1% 66.7% 61.5% CNN AE 71.0% 72.0% 71.0% BiLSTM(a) 77.8% 78.8% 78.3% FF-DDE 82.8% 83.0% 82.4% DF-DDE 80.8% 80.9% 80.8% LSTM with Gating 80.0% 80.9% 81.0% SCAED 84.8% 85.1% 84.9% Based on Table 1, the experimental results of the multimodal depression recognition model SCAED can be obtained. Among them, the experimental results of the multimodal depression recognition model SCAED are as follows: the precision of the multimodal depression recognition model SCAED is 84.8%, the recall rate is 80.9%, and the F1 value is 84.9%.

[0061] Then, in order to verify the effectiveness of each modal feature and emotional feature, the present invention conducts emotional ablation experiments and modal ablation experiments. The experimental results are shown in Table 2.

[0062] Among them, SCAED (t) means that the model only uses text features and removes text emotional information for depression identification; SCAED (te) means that the model uses text features for depression identification (including text semantic features and text emotional features); SCAED (a) means that the model only uses speech features and removes speech emotional information for depression identification; SCAED (ae) means that the model only uses speech features for depression identification (including speech deep features and speech emotional features); SCAED ( / e) means that the model only removes emotional information for depression identification; SCAED (full) means that the complete model is used for depression identification.

[0063] Table 2 Results of emotion ablation experiment and modality ablation experiment model Precision Recall F1 SCAED(t) 70.0% 72.5% 71.4% SCAED(t-e) 80.0% 80.9% 80.4% SCAED(a) 73.3% 74.6% 73.9% SCAED(a-e) 77.8% 78.8% 78.3% SCAED( / e) 65.4% 70.2% 67.7% SCAED(full) 84.8% 85.1% 84.9% Experimental results show that multimodal data can complement information and is better than depression detection based on unimodal data; the addition of emotional information is helpful for both unimodal and multimodal depression detection and can effectively improve the performance of the model.

[0064] Next, in order to verify the effectiveness of the multi-factor fusion part, the present invention will add the multi-factor fusion module to the multimodal depression recognition model SCAED to obtain a complete multimodal multi-factor model GCM-SCAED. Table 3 shows the results of the GCM-SCAED comparison experiment and ablation experiment.

[0065] Table 3 GCM-SCAED comparison experiment and ablation experiment results model Precision Recall F1 baseline model 60.0% 43.0% 50.0% FS-GSM 78.0% 63.5% 70.0% Two BiLSTM 78.0% 80.0% 80.0% LSTMwithGating 80.0% 80.9% 81.0% FF-DDE 82.8% 83.0% 82.4% DF-DDE 80.8% 80.9% 80.8% Trf+CNN 91.0% 83.0% 87.0% GRU / BiLSTM 79.0% 92.0% 85.0% SCAED 84.8% 85.1% 84.9% GCM-SCAED (M) 85.7% 93.8% 89.7% GCM-SCAED (F) 100.0% 84.7% 92.4% GCM-SCAED 92.3% 89.2% 90.3% The comparative experimental results show that the GCM-SCAED model achieved the best results in the F1 index on the DAIC-WOZ dataset, and the male and female models achieved the best results in precision and recall respectively.

[0066] Then, in order to verify the effectiveness of the multi-factor part, the ablation experiment part of the present invention will conduct experiments on male and female models respectively. The experimental results of the ablation experiment part of the male model are shown in Table 4, among which, the male model GCM-SCAED (M): (1) GCM-SCAED (Mt) means that the male model only uses text semantic features for depression recognition; (2) GCM-SCAED (Mte) means that the male model uses text features (including text semantic features and text emotional features) for depression recognition; (3) GCM-SCAED (Ma) means that the male model only uses deep speech features for depression recognition; (4) GCM-SCAED (Mae) means that the male model uses speech features (including deep speech features and speech emotional features) for depression recognition; (5) GCM-SCAED (M / C) means that the male model removes common sense knowledge; (6) GCM-SCAED (M) means that the model is a complete male depression recognition model.

[0067] Table 4 Experimental results of the male model ablation experiment model Precision Recall F1 GCM-SCAED (M-t) 33.3% 90.1% 62.0% GCM-SCAED (M-t-e) 60.0% 85.7% 70.6% GCM-SCAED (M-a) 40.0% 85.7% 54.5% GCM-SCAED (M-a-e) 45.5% 71.4% 55.6% GCM-SCAED (M / C) 82.5% 87.6% 85.1% GCM-SCAED (M) 85.7% 93.8% 89.7% The experimental results of the female model ablation experiment are shown in Table 5, where GCM-SCAED (F) is the female model: (1) GCM-SCAED (Ft) indicates that the female model only uses text semantic features for depression recognition; (2) GCM-SCAED (Fte) indicates that the female model uses text features (including text semantic features and text emotional features) for depression recognition; (3) GCM-SCAED (Fa) indicates that the female model only uses deep speech features for depression recognition; (4) GCM-SCAED (Fae) indicates that the female model uses speech features (including deep speech features and speech emotional features) for depression recognition; (5) GCM-SCAED (F / C) indicates that the female model removes common sense knowledge; (6) GCM-SCAED (F) indicates that the model is a complete female depression recognition model.

[0068] Table 5 Experimental results of the ablation experiment of the female model model Precision Recall F1 GCM-SCAED (F-t) 66.7% 57.1% 61.5% GCM-SCAED (F-t-e) 63.6% 78.9% 71.3% GCM-SCAED (F-a) 46.8% 68.9% 57.4% GCM-SCAED (F-a-e) 54.5% 85.7% 66.7% GCM-SCAED (F / C) 91.5% 81.4% 86.6% GCM-SCAED (F) 100.0% 84.7% 92.4% In summary, the effect achieved by the GCM-SCAED model is better than the existing optimal model, and it achieves the best results in terms of F1 and Precision. At the same time, relevant results show that men and women have different characteristics. Considering gender factors in depression detection and building different classification models to train them separately can help accurately identify depressed patients. Secondly, introducing a common sense knowledge base for reasoning in depression detection and adding knowledge related to user moods and feelings can help to gain a deeper understanding of user thoughts and perceive changes in user moods, thereby accurately distinguishing between depressed patients and non-depressed users and improving the accuracy of the model.

[0069] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure to form a technical solution.< / sync> < / fam> < / an>

Claims

1. A multi-modal and multi-factor depression recognition system integrating emotional information, characterized in that: It includes a preprocessing module, a text feature extraction module, a speech feature extraction module, a multimodal fusion module and a multi-factor fusion module; The preprocessing module is used to preprocess the original speech of the user to be detected, and obtain the preprocessed text as a clause and a speech waveform graph, wherein the abscissa of the speech waveform graph is time, and the ordinate of the speech waveform graph is amplitude; The text feature extraction module is used to add [CLS] at the start position of the preprocessed clause and [SEP] at the end position, segment the clause after adding the marker through a marker to obtain segmented clauses; extract features from the segmented clauses through BERT to obtain semantic features. f ts ; The segmented clauses are processed through the text sentiment extraction model to obtain the text sentiment features f te , the semantic features f ts and the text sentiment features f te Integrate into text features f t ; The text emotion extraction model is obtained in the following way: Obtain a clause in the SST-2 data set as a first input sample, and use the emotion label corresponding to the clause in the SST-2 data set as a first output sample, wherein the first input sample and the first output sample constitute a first training sample, and perform model training on the RoBERTa model based on multiple first training samples until the RoBERTa model converges to obtain a trained RoBERTa model, retain the parameters and encoder structure of the trained RoBERTa model, and replace the output layer of the trained RoBERTa model with a fully connected layer FC to obtain a text emotion extraction model; The speech feature extraction module is used to extract features from the speech waveform to obtain speech deep features. f ad , input the original speech into the speech emotion extraction model to obtain the speech emotion features f ae , the deep features of speech f ad and voice emotion features f ae Concatenate to get complete speech features f a ; The speech emotion extraction model is obtained by training the Wav2Vec2.0 model based on multiple second training samples, wherein the second training samples include a second input sample and a second output sample, the second input sample is a speech sample in the IEMOCAP data set, and the second output sample is a speech emotion feature corresponding to the speech sample in the IEMOCAP data set; The multimodal fusion module is used to f t and voice features f a , generate text modality features S t , intra-modal features of speech S a , features in speech associated with text S ta and features in text that are associated with speech S at , according to the text modality features S t , intra-modal features of speech S a , features in speech associated with text S ta and features in text that are associated with speech S at , calculate the first fusion multimodal feature and the second fusion multimodal features ; The multi-factor fusion module is used to determine the gender of the user to be detected. When the user to be detected is male, the first fused multimodal feature is classified by a male-oriented classification model. Processing is performed to obtain the depression prediction result of the user to be detected. When the user to be detected is a female, the second fusion multimodal feature is classified by the female-oriented classification model. Processing is performed to obtain the depression prediction result of the user to be detected.

2. According to claim 1, a multi-modal and multi-factor depression recognition system integrating emotional information is characterized in that: The preprocessing module is specifically used to obtain the original speech of the user to be detected, determine the triple <start time, end time, clause> of the user to be detected in the original speech, use regular expressions to clean the clauses, and obtain preprocessed clauses; perform pre-emphasis processing on the original speech to obtain pre-emphasized speech, specifically, enhance the high-frequency part of the original speech, improve the high-frequency resolution of the original speech, perform frame processing on the pre-emphasized speech, apply a Hanning window function to the framed speech to enhance the speech signal, obtain an enhanced speech signal, and draw a speech waveform diagram according to the enhanced speech signal.

3. The multi-modal and multi-factor depression recognition system integrating emotional information according to claim 1 is characterized in that: The text feature extraction module uses BERT to extract features from the segmented clauses to obtain semantic features. f ts ,include: Get word embedding, position embedding and tag embedding in the segmented clauses, add the word embedding, position embedding and tag embedding to get the added embedding, input the added embedding into the Transformer encoder for encoding to get the semantic features f ts .

4. The multi-modal and multi-factor depression recognition system integrating emotional information according to claim 1, characterized in that: The speech feature extraction module extracts features from the speech waveform to obtain speech deep features. f ad ,include: Grayscale the speech waveform to obtain a grayscale waveform, normalize and standardize the grayscale waveform to obtain a processed waveform, wherein the pixel values ​​in the processed waveform are within the range of [0, 1], and the pixel values ​​have a uniform mean and standard deviation, and the processed waveform is converted into a one-dimensional matrix and input into TCN to obtain deep features of speech. f ad .

5. The multi-modal and multi-factor depression recognition system integrating emotional information according to claim 1, characterized in that: The multimodal fusion module is specifically used to combine text features f t Input into the self-attention Transformer to obtain the text modality features S t , which is specifically achieved through the following formula: ； ； ； ； in, W Qt , W Kt , W Vt is the parameter matrix, T represents the transpose of a matrix, d k is the vector dimension, sum It means sum; The voice features f a Input into the self-attention Transformer to obtain the features within the speech modality S a , which is specifically achieved through the following formula: ； ； ； ； in, W Qa , W Ka , W Va is the parameter matrix; Through the cross attention mechanism, according to the features in the text modality S t and intra-modal features of speech S a , calculate the features in speech associated with text S ta , expressed by the following formula: ； Through the cross attention mechanism, according to the features in the text modality S t and intra-modal features of speech S a , calculate the features associated with speech in the text S at , expressed by the following formula: ； Features in text modality S t , intra-modal features of speech S a , features in speech associated with text S ta Features associated with speech S at Perform concatenation operation to obtain matrix U M =[ S t , S a , S ta , S at ], the matrix U M =[ S t , S a , S ta , S at ], input two layers Transformer Encoder In the first fusion multimodal feature , which is specifically expressed by the following formula: =Transformer ( U M )； Features in text modality S t , intra-modal features of speech S a , features in speech associated with text S ta Features associated with speech S at Perform the sum operation and get the matrix U F =[ S t , S a , S ta , S at ], the matrix U F =[ S t , S a , S ta , S at ], input two layers Transformer Encoder In the second fusion multimodal feature , which is specifically expressed by the following formula: =Transformer ( U F )。 6. A multi-modal and multi-factor depression recognition system integrating emotional information according to claim 5, characterized in that: Cross-Attention ( S t , S a ) is calculated by the following formula: ； ； ； ； Similarly, Cross-Attention ( S a , S t ) is calculated by the following formula: ； ； ； 。 7. The multi-modal and multi-factor depression recognition system integrating emotional information according to claim 1, characterized in that: The multi-factor fusion module uses a male-oriented classification model to classify the first fused multimodal features. Processing is performed to obtain the depression prediction results of the user to be tested, including: Acquire inferences about emotions and feelings xReact , based on the common sense knowledge base COMET, the preprocessed clauses and reasoning relations xReact Processing is performed to obtain common sense knowledge, which is specifically achieved through the following formula: ； in, s ti The first clause in the preprocessed i words, s ki For the i Common sense knowledge corresponding to each word, ⊕ indicates concatenation; Common sense knowledge s ki , input into BERT to encode and get common sense knowledge s ki The embedding representation of b i , which is specifically achieved through the following formula: ； The embedding representation of all common sense knowledge forms the sentence vector B = { b 1, b 2,…, b m }, input the sentence vector B Transformer , get the target common sense knowledge K u , achieved through the following formula: ； Target common sense knowledge K u And the first fusion multimodal features Splice and get the first feature , the first feature enter BiGRU In the classifier, we get features containing forward and backward information , which is specifically expressed by the following formula: ； The features that will contain forward and backward information Input the fully connected layer to obtain the depression prediction result of the user to be tested y’ , which is specifically expressed by the following formula: 。 8. The multi-modal and multi-factor depression identification system integrating emotional information according to claim 1, characterized in that: The multi-factor fusion module uses a female-oriented classification model to fusion the second multi-modal feature Processing is performed to obtain the depression prediction results of the user to be tested, including: Acquire inferences about emotions and feelings xReact , based on the common sense knowledge base COMET, the preprocessed clauses and reasoning relations xReact Processing is performed to obtain common sense knowledge, which is specifically achieved through the following formula: ; in, s ti The first clause in the preprocessed i words, s ki For the i Common sense knowledge corresponding to each word; Common sense knowledge s ki , input into BERT to encode and get common sense knowledge s ki The embedding representation of b i , which is specifically achieved through the following formula: ; The embedding representation of all common sense knowledge forms the sentence vector B = { b 1, b 2,…, b m }, input the sentence vector B Transformer , get the target common sense knowledge K u , achieved through the following formula: ; Target common sense knowledge K u and the second fusion multimodal features Splice and get the second feature , the second feature Input into the MLP classifier to obtain the depression prediction result of the user to be tested y’ , which is specifically expressed by the following formula: 。 9. A multi-modal and multi-factor depression recognition system integrating emotional information according to claim 8, characterized in that: The specific calculation is achieved through the following formula: ; ; ; in, , and is the parameter matrix, , and is the bias parameter, is the output of the first hidden layer, is the output of the second hidden layer.

10. The multi-modal and multi-factor depression recognition system integrating emotional information according to claim 1, characterized in that: The multimodal and multifactor depression recognition system integrating emotional information further includes a loss module; The loss module is used to use the cross entropy loss function as the loss function, which is specifically expressed by the following formula: ； in, For the j The actual results of depression in samples For the j Depression prediction results of samples; The value of the loss function is used to update the parameters in the preprocessing module, text feature extraction module, speech feature extraction module, multimodal fusion module and multi-factor fusion module.

Images