Depression detection method and system based on emotional expression

Abstract

The application relates to the technical field of information technology, and discloses a depression detection method and system based on emotional expression. Specifically disclosed are the following: obtaining voice data and text data from an original doctor-patient conversation voice segment, extracting voice features and text features by using a pre-training model, and automatically labeling voice emotion and text emotion based on the same. Then, voice feature embedding and text feature embedding are extracted, multi-head attention mechanism is used to extract emotional representations of the voice and the text, cross-attention mechanism is used to analyze the degree of mismatch between the voice and the text emotional representations, and emotional inconsistency feature embedding is obtained. The voice feature embedding, the text feature embedding and the emotional inconsistency feature embedding are fused, and depression categories are predicted according to the fused features. The automatic emotional labeling method improves the labeling efficiency; the phenomenon of inconsistent emotional expression is used to assist depression diagnosis, and the accuracy of depression detection is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, in particular to a depression detection method and system based on emotional expression. BACKGROUND

[0002] Major Depressive Disorder (MDD) is one of the most common mental disorders, which is a kind of mood disorder caused by various reasons, with significant and persistent low mood as the main clinical feature, accompanied by varying degrees of cognitive and behavioral changes, and some patients have self-injury, suicidal behavior, and even death.

[0003] Due to the serious harm and complexity of depression detection, there are many relevant researches in the field of depression detection. Researchers use various modal data, including speech, text and multi-modal fusion, to improve the recognition accuracy.

[0004] In terms of speech, the complexity of speech production makes the human articulatory system a sensitive output system. Even slight physiological and cognitive changes can cause significant changes in voice. It has been proven that cognitive impairment related to depression affects the working memory of patients, which directly changes the speech production, and depression also affects the body and autonomic nervous system, which may change the mechanism involving speech production, voice production and articulatory muscle system.

[0005] In terms of text, in recent years, the maturity of natural language processing technology has made related research based on text data popular. Psychologists analyze the language and writing text of patients with depression and find that there are differences in the expression of words between patients with depression and healthy people.

[0006] In terms of multi-modal data fusion, multi-modal fusion technology can analyze and process data from different sources and different forms, providing more comprehensive and valuable information for the final decision of the depression recognition model, so as to improve the accuracy of depression recognition.

[0007] Emotional consistency, i.e., whether an individual exhibits consistent emotional states across different modalities (e.g., speech and text), is an important dimension for mood disorder detection. For patients with depression, they may try to conceal their true emotions in speech, but inadvertently reveal their true emotional states in text expression, and vice versa. Therefore, by analyzing the emotional consistency of patients in both speech and text modalities, it is possible to provide a new perspective and method for the automated detection of depression. Currently, there are still limited studies on the use of emotional consistency in speech and text modalities for the automated detection of depression. Existing techniques and methods fail to effectively integrate and analyze emotional information in multi-modal data, especially lacking a framework and algorithm that can systematically evaluate the emotional consistency of speech and text modalities. In addition, how to accurately extract emotional features from a large amount of unstructured speech and text data, and how to establish the correlation between emotional consistency and depression detection, are technical problems that existing techniques have not solved. SUMMARY

[0008] The present application provides a depression detection method based on emotional expression to solve the problems in the prior art of how to effectively integrate and analyze emotional information in multi-modal data, how to accurately extract emotional features from a large amount of unstructured speech and text data, and how to establish the correlation between emotional consistency and depression detection.

[0009] Correspondingly, the present application also provides a depression detection system based on emotional expression, an electronic device, and a computer readable storage medium, for ensuring the implementation and application of the above method.

[0010] To solve the above technical problems, the present application discloses a depression detection method based on emotional expression, the method comprising:

[0011] obtaining speech data and text data of a target speaker from an original doctor-patient conversation speech segment;

[0012] extracting speech features in the speech data and text features in the text data using a pre-trained model, and automatically labeling speech emotion and text emotion according to the speech features and the text features;

[0013] extracting speech feature embeddings and text feature embeddings from the speech features and the text features, respectively;

[0014] extracting emotional representations from the speech features and the text features using a multi-head attention mechanism, and analyzing the degree of mismatch between the emotional representations in the speech features and the text features using a cross-attention mechanism to obtain emotional inconsistency feature embeddings;

[0015] fusing the speech feature embeddings, the text feature embeddings, and the emotional inconsistency feature embeddings, and predicting a depression category according to the fused features.

[0016] Preferably, the voice data and the text data of the target speaker are obtained from the original doctor-patient conversation voice segment, comprising:

[0017] The speaker separation and identification are performed on the original doctor-patient conversation voice segment using the speaker diarization technology, and the voice data of the target speaker is obtained;

[0018] The voice data of the target speaker is recognized from the original doctor-patient conversation voice segment using the automatic speech recognition technology, and the text data of the target speaker is obtained.

[0019] Preferably, the pre-trained model comprises a HuBERT model and a RoBERTa model;

[0020] The voice features in the voice data and the text features in the text data are extracted using the pre-trained model, and the automatic annotation of the voice emotion and the text emotion is performed according to the voice features and the text features, comprising:

[0021] The voice features in the voice data are extracted using the HuBERT model, and the voice emotion classification is performed using the Transformer model;

[0022] The text features in the text data are extracted using the RoBERTa model, and the text emotion classification is performed using the Transformer model.

[0023] Preferably, the voice feature embedding and the text feature embedding are extracted from the voice features and the text features, respectively, comprising:

[0024] The voice features and the text features are input into multiple Transformer blocks, and the voice feature embedding and the text feature embedding are obtained after being processed by the average pooling layer.

[0025] Preferably, the emotion representation is extracted from the voice features and the text features using the multi-head attention mechanism, and the degree of mismatch between the emotion representations in the voice features and the text features is analyzed using the cross-attention mechanism, and the emotion inconsistency feature embedding is obtained, comprising:

[0026] The voice features and the text features are input into multiple Transformer blocks, and the voice emotion features and the text emotion features are obtained;

[0027] After the voice emotion features and the text emotion features are linearly changed, the emotion inconsistency feature embedding is obtained through the cross-attention layer;

[0028] Wherein, the Transformer block uses the multi-head attention mechanism to extract the voice emotion features and the text emotion features; the cross-attention layer uses the cross-attention mechanism to obtain the emotion inconsistency feature embedding.

[0029] Preferably, the voice emotion features and the text emotion features are linearly changed, and then the emotion inconsistency feature embedding is obtained through the cross-attention layer, including:

[0030] The voice emotion features and the text emotion features are linearly changed, and then the first representation vector and the second representation vector of the inconsistency degree between the voice emotion expression and the text emotion expression are obtained through the cross-attention layer;

[0031] The voice emotion features, the text emotion features, the first representation vector and the second representation vector are spliced to obtain a spliced vector;

[0032] The spliced vector is input into a full connection layer to determine whether the emotion expression is consistent;

[0033] The hidden output in the cross-attention layer is taken as the emotion inconsistency feature embedding.

[0034] The application also discloses a depression detection system based on emotion expression, which comprises:

[0035] A data preprocessing module is configured to obtain voice data and text data of a target speaker from an original doctor-patient dialogue voice segment;

[0036] An automatic labeling module is configured to extract voice features from the voice data and text features from the text data by using a pre-trained model, and automatically label voice emotion and text emotion according to the voice features and the text features;

[0037] A voice and text feature extraction module is configured to extract voice feature embedding and text feature embedding from the voice features and the text features respectively;

[0038] An emotion inconsistency feature extraction module is configured to extract emotion representation from the voice features and the text features by using a multi-head attention mechanism, and analyze the mismatching degree between the emotion representation in the voice features and the text features by using a cross-attention mechanism, so as to obtain emotion inconsistency feature embedding;

[0039] A result prediction module is configured to fuse the voice feature embedding, the text feature embedding and the emotion inconsistency feature embedding, and predict a depression category according to the fused features.

[0040] Preferably, the data preprocessing module comprises:

[0041] A first data preprocessing submodule is configured to separate and identify speakers of the original doctor-patient dialogue voice segment by using a speaker log technology, so as to obtain voice data of a target speaker;

[0042] A second data preprocessing submodule is configured to recognize the voice data of the target speaker by using an automatic speech recognition technology, so as to obtain text data of the target speaker.

[0043] The application also discloses an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements one or more methods described in the application when executing the program.

[0044] The application also discloses a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement one or more methods described in the application.

[0045] The application has the following beneficial effects:

[0046] (1) The concept of emotional inconsistency is first introduced, and the difference in emotional expression of depression patients in speech and text modalities is compared to assist in improving the reliability of depression detection.

[0047] (2) The multi-head self-attention mechanism is used to extract emotional representations from the sound and text fields, and then the cross-attention mechanism is used to obtain the degree of mismatch between emotional expressions in sound and text data, so as to effectively obtain emotional inconsistency information highly related to depression.

[0048] (3) By fusing speech emotional information, text emotional information and speech-text emotional expression inconsistency information, the accuracy of depression detection is increased, and the possible information omission in the prior art is improved.

[0049] (4) More accurate speaker separation and identification technology is adopted, and the speech signal of the depression patient is specially processed to improve the quality and availability of the speech data.

[0050] (5) An automatic emotional labeling method is adopted to improve the labeling efficiency and reduce the cost and error of manual labeling.

[0051] Additional aspects and advantages of the application will be described in the following description section, which will become apparent from the following description or be understood through practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0052] The above and / or additional aspects and advantages of the application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0053] Figure 1 A flowchart of a depression detection method based on emotional expression provided for the embodiments of the application is shown in the figure;

[0054] Figure 2 A speech data preprocessing flowchart provided for the embodiments of the application is shown in the figure;

[0055] Figure 3 A voice data automatic emotion labeling flowchart provided for an embodiment of the present application;

[0056] Figure 4 A text data automatic emotion labeling flowchart provided for an embodiment of the present application;

[0057] Figure 5 A structure schematic diagram of a voice-text emotion expression inconsistency information extraction module provided for an embodiment of the present application;

[0058] Figure 6 An automatic depression detection network framework diagram based on emotion expression inconsistency provided for an embodiment of the present application;

[0059] Figure 7 A structure schematic diagram of a depression detection system based on emotion expression provided for an embodiment of the present application;

[0060] Figure 8 A structure schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0061] Embodiments of the present application are described in detail below with reference to examples illustrated in the accompanying drawings, in which the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0062] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the phrase "comprising" used in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0063] As will be understood by one of skill in the art, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs unless otherwise defined herein. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0064] The scheme provided by the embodiments of the present application can be executed by any electronic device, which can be a terminal device or a server. The server can be a physical server, a server cluster composed of multiple physical servers, a distributed system, or a cloud server providing cloud computing services. The terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the present application. For the technical problems existing in the prior art, the method and system for obtaining speech data and text data of a target speaker from an original doctor-patient conversation speech segment provided by the present application aims to solve at least one of the technical problems of the prior art.

[0065] The technical scheme of the present application and how the technical scheme of the present application solves the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0066] The embodiments of the present application provide a depression detection method based on emotional expression, which aims to introduce the concept of emotional inconsistency analysis and improve the accuracy and efficiency of depression detection by comprehensively considering the emotional expression of individuals in speech and text modalities.

[0067] As shown in Figure 1 The method can include the following steps:

[0068] Step 101, obtaining speech data and text data of a target speaker from an original doctor-patient conversation speech segment.

[0069] Step 102, extracting speech features in the speech data and text features in the text data using a pre-trained model, and automatically labeling speech emotion and text emotion according to the speech features and the text features.

[0070] The pre-training model in the embodiments of the present application is obtained by training a large amount of data, and two different pre-training models can be used to extract speech features in speech data and text features in text data respectively, and the speech features are used for speech emotion classification, and the text features are used for text emotion classification, and then the speech emotion annotation and the text emotion annotation are obtained.

[0071] In step 103, speech feature embeddings and text feature embeddings are extracted from the speech features and the text features respectively.

[0072] By extracting speech feature embeddings from speech features and text feature embeddings from text features, complex temporal, local and global dependencies related to emotional expression in speech data and text data can be captured, thereby improving the performance and generalization ability of the model.

[0073] In step 104, a multi-head attention mechanism is used to extract emotional representations from speech features and text features, and a cross-attention mechanism is used to analyze the degree of mismatch between emotional representations in speech features and text features, and obtain emotional inconsistency feature embeddings.

[0074] Depression patients usually show special emotional expression in their speech. For example, depression patients may convey negative emotional content in an unexpected calm manner, showing a high degree of inconsistency in emotional expression. These unique phenomena can be used as valuable indicators for assessing symptoms. Therefore, the embodiments of the present application use this inconsistency phenomenon to improve the accuracy of depression diagnosis.

[0075] In step 105, the speech feature embeddings, the text feature embeddings and the emotional inconsistency feature embeddings are fused, and the depression category is predicted according to the fused features.

[0076] By combining the emotional inconsistency feature embeddings with the identified speech feature embeddings and text feature embeddings in the speech and text fields, emotional expression inconsistency information highly related to depression can be effectively obtained.

[0077] In the embodiments of the present application, the voice data and the text data of the target speaker are obtained from the original doctor-patient conversation voice segment, the voice features in the voice data and the text features in the text data are extracted by using a pre-trained model, and the voice emotion and the text emotion are automatically labeled according to the voice features and the text features. The automatic emotion labeling method is used to improve the labeling efficiency and reduce the cost and error of manual labeling. Then, the voice feature embedding and the text feature embedding are extracted from the voice features and the text features respectively. The emotion representation is extracted from the voice features and the text features by using the multi-head attention mechanism, and the degree of mismatch between the emotion representations in the voice features and the text features is analyzed by using the cross-attention mechanism to obtain the emotion inconsistency feature embedding. The concept of emotion inconsistency is introduced, and the difference in emotional expression of the depression patient in the voice and the text is compared to effectively capture the emotion inconsistency information highly related to depression, which can assist in improving the reliability of depression detection. Finally, the voice feature embedding, the text feature embedding and the emotion inconsistency feature embedding are fused, and the depression category is predicted according to the fused features, which effectively integrates and analyzes the emotion information in the multi-modal data, increases the accuracy of depression detection, and improves the possible information omission or conflict in the prior art.

[0078] Based on the above, the embodiments of the present application provide a scientific and effective method for early identification and intervention of depression, especially for those mild or early depression patients who are difficult to be accurately detected in traditional detection methods, which improves the accuracy of depression detection, improves the efficiency and user experience of the automatic depression detection system, simplifies the operation process by establishing an automatic process, and makes the detection of depression more convenient and efficient.

[0079] In an optional embodiment, the voice data and the text data of the target speaker are obtained from the original doctor-patient conversation voice segment, including:

[0080] The speaker separation and identification of the original doctor-patient conversation voice segment are performed by using the speaker log technology to obtain the voice data of the target speaker;

[0081] The voice data of the target speaker is recognized by using the automatic speech recognition technology to obtain the text data of the target speaker.

[0082] The original doctor-patient conversation voice segment contains the doctor's inquiry and the patient's feedback, and is more likely to reflect the emotional expression of the patient. In order to accurately analyze the patient's voice and emotional expression, it is necessary to first separate the doctor's voice from the patient's voice. In the embodiments of the present application, the open source toolkit pyannote-audio is used to automatically segment each audio file of the original doctor-patient conversation voice, and then the patient's voice segment is extracted as the voice data of the target speaker. Subsequently, the patient's voice segment is sent to an automatic speech recognition (ASR) system, and the WeNet2 toolkit is used to accurately convert the patient's voice into text, thereby obtaining the text data of the target speaker, which provides a basis for subsequent emotional analysis.

[0083] As shown in Figure 2 , the original doctor-patient conversation voice segment corresponds to an audio signal input into the open source toolkit pyannote-audio, which uses speaker diarization technology to segment the input audio signal into a voice segment of speaker 1 and a voice segment of speaker 2, and performs speech recognition on the voice segment of speaker 1 and the voice segment of speaker 2 through an ASR system, and finally obtains the voice and text of speaker 1 and the voice and text of speaker 2.

[0084] In the embodiments of the present application, since the voice features of the depression patients may change due to the influence of emotions, a more accurate speaker diarization technology is used to specially process the voice data of the depression patients, thereby improving the quality and availability of the voice data.

[0085] In an optional embodiment, the pre-training model includes a HuBERT model and a RoBERTa model.

[0086] The pre-training model is used to extract voice features in the voice data and text features in the text data, and automatically label the voice emotion and the text emotion according to the voice features and the text features, including:

[0087] The HuBERT model is used to extract voice features in the voice data, and the Transformer model is used for voice emotion classification.

[0088] The RoBERTa model is used to extract text features in the text data, and the Transformer model is used for text emotion classification.

[0089] The emotional inconsistency concept is introduced in the embodiments of the present application. Extracting emotional expression inconsistency information of depression patients needs to rely on emotional labeling information in the speech and text space. However, the traditional method needs manual labeling, which is not only time-consuming and labor-intensive, but also the labeling result may have subjectivity and inconsistency. In order to solve this problem, the embodiments of the present application provide an automatic emotional labeling method based on HuBERT model and RoBERTa model.

[0090] As shown in Figure 3 For speech data, first perform feature extraction based on a self-supervised learning pre-trained model, and then use a Transformer model to perform speech emotion classification, that is, divide the speech data into negative emotion, neutral emotion and positive emotion. In the automatic emotion labeling algorithm based on speech, speech features in the speech data are extracted by using HuBERT (Hidden Unit BERT). HuBERT is a revolutionary self-supervised speech processing model designed based on the idea of BERT architecture, which is specially used to learn deep speech representations from a large amount of unlabeled audio data. By imitating the mask language model training method of BERT, HuBERT uses pseudo-labels generated by a clustering algorithm to predict the category of the masked part in the audio segment during the pre-training phase. This training method enables HuBERT to capture complex patterns and details in speech data without explicit labeling, such as key features such as pitch, rhythm and speech texture. In addition, HuBERT not only improves the accuracy of speech recognition, but also deeply analyzes the emotional and semantic aspects of speech. Therefore, in the embodiments of the present application, the HuBERT model is used to extract information related to speech emotion for speech emotion classification. Based on the above, automatic emotion labeling of speech is realized.

[0091] As shown in Figure 4As shown, for text data, first feature extraction based on self-supervised learning pre-training model is performed, and then a Transformer model is used for speech emotion classification, i.e., text data is classified into negative emotion, neutral emotion, and positive emotion. In the text-based automatic emotion labeling algorithm, RoBERTa (Robustly Optimized BERT Approach) is used to extract original text features. RoBERTa is an advanced self-supervised natural language processing model that further optimizes the BERT architecture. RoBERTa significantly enhances the model's ability to process text data through several key improvements to the BERT training strategy, increasing the depth of the model's understanding of text. RoBERTa not only performs well on basic language model tasks, but also shows extremely high effectiveness in downstream applications such as sentiment analysis, text classification, and question-answering systems. Therefore, in the embodiments of the present application, RoBERTa model is used to extract information related to text sentiment for text sentiment classification. Based on the above, automatic emotion labeling of text is realized.

[0092] After the automatic emotion labeling of speech and text, the speech and text emotion classification of each segment can be obtained, which lays a foundation for subsequent extraction of speech and text modal emotion inconsistency information.

[0093] In an optional embodiment, the speech feature embedding and the text feature embedding are extracted from the speech feature and the text feature, respectively, including:

[0094] The speech feature and the text feature are input into multiple Transformer blocks, and after being processed by an average pooling layer, the speech feature embedding and the text feature embedding are obtained.

[0095] In an optional embodiment, the emotion representation is extracted from the speech feature and the text feature using a multi-head attention mechanism, and the degree of mismatch between the emotion representations in the speech feature and the text feature is analyzed using a cross-attention mechanism to obtain an emotion inconsistency feature embedding, including:

[0096] The speech feature and the text feature are input into multiple Transformer blocks to obtain speech emotion features and text emotion features; wherein the Transformer block uses a multi-head attention mechanism to extract the speech emotion features and the text emotion features.

[0097] As Figure 5As shown, this embodiment of the application extracts the emotional inconsistency feature embedding through a speech-text emotional expression inconsistency information extraction module. The supervision labels of this module are emotionally consistent (marked as "1") and emotionally inconsistent (marked as "0"). That is, when a speech segment and its corresponding text have the same emotional label, the supervision label corresponding to this speech-text segment is set to "1", otherwise it is set to "0".

[0098] Figure 5 Input speech features (Extracted by Hubert module) and text features (Extracted by the RoBERTa module) Contains general information such as speech speed, intonation, syntactic structure, and text semantics. This information is closely related to emotional expression. Speech Feature X (A) and text features X (T) The speech emotion features X′ are obtained by using N Transformer blocks respectively. (A) and text sentiment features X′ (T) It is possible to obtain complex local and global temporal dependencies related to emotional expression in both the speech and text domains.

[0099] After linear transformation of speech and text sentiment features, sentiment inconsistency feature embeddings are obtained through a cross-attention layer; the cross-attention layer uses a cross-attention mechanism to obtain sentiment inconsistency feature embeddings.

[0100] In an optional embodiment, after linearly transforming the speech emotion features and text emotion features, an emotion inconsistency feature embedding is obtained through a cross-attention layer, including:

[0101] After linear transformation of speech emotion features and text emotion features, the first and second representation vectors of the degree of inconsistency between speech emotion expression and text emotion expression are obtained through a cross-attention layer.

[0102] Voice emotion feature X′ (A) Q is obtained after linear transformation. (A) K (A) V (A) Similarly, the text sentiment feature X′ (T) Q is obtained after linear transformation. (T) K (T) V (T) By using a cross-attention layer, a first representation vector X is obtained to represent the degree of inconsistency between textual and auditory emotional expression. (AT) Second representation vector X (TA) :

[0103]

[0104]

[0105] concatenate the voice emotion feature, the text emotion feature, the first representation vector and the second representation vector to obtain a concatenated vector;

[0106] Specifically, the voice emotion feature, the text emotion feature, the first representation vector and the second representation vector are respectively subjected to Average Pooling processing and then concatenated (Concat) to obtain a concatenated vector h (E) :

[0107] h (E) =[Avg(X′ (A) );Avg(X (AT) );Avg(X (TA) );Avg(X′ (T) )]

[0108] input the concatenated vector into a fully connected layer to determine whether the emotional expression is consistent;

[0109] In the embodiments of the present application, the concatenated vector is input into three fully connected layers (FC1, FC2 and FC3) to distinguish whether the emotional expression of the current cross-voice and text mode is consistent.

[0110] The hidden output in the cross-attention layer is taken as an emotional inconsistency feature embedding.

[0111] In the embodiments of the present application, the hidden output e (E) in the cross-attention layer is used to represent the emotional expression inconsistency information, which is usually called "embedding" representation, i.e. emotional inconsistency feature embedding. Since the hidden output contains abnormal emotional expression patterns in the voice and text fields, the obtained emotional inconsistency feature embedding inherently captures the basic information to distinguish the alignment of emotional expression in these fields. This ability has important value for downstream tasks, especially in the field of depression detection.

[0112] e (E) =FC(h (E) )

[0113] In an optional embodiment, an automatic depression detection network based on emotional expression inconsistency is proposed for the implementation of the above method. As shown in Figure 6As shown, the network comprises a main network for depression detection and a sub-network for judging emotional inconsistency. The main network uses speech data, text data, and emotional inconsistency information extracted from the sub-network to detect depression, while the sub-network focuses on judging whether the emotional expression in speech and text is consistent. The sub-network aims to extract emotional inconsistency information from the main network, thereby improving the accuracy of depression detection by the main network and automating depression detection.

[0114] First, HuBERT and RoBERTa feature extractions are performed on the speech and text data respectively to obtain speech features X. (A) and text features X (T) The main network then feeds the extracted speech and text features into a series of Transformer blocks to achieve Transformer-based feature aggregation, capturing complex dependencies in the data. The speech feature embedding obtained after multiple Transformer layers and average pooling layers is then processed. (a) and text feature embedding e (t) The sub-network will extract the speech features X (A) and text features X (T) The emotion inconsistency information extraction module extracts the emotion inconsistency feature embedding e. (E) After feature fusion, the feature representation e is obtained. (fusion) :

[0115] e (fusion) =[e (a) ;e (E) ;e (t) ]

[0116] Finally, a prediction of the depression category is generated through a fully connected layer.

[0117] Based on the same principles as the methods provided in the embodiments of this application, the embodiments of this application also provide a depression detection system based on emotion expression, such as... Figure 7 As shown, the system includes:

[0118] The data preprocessing module 701 is used to obtain the speech data and text data of the target speaker from the original doctor-patient dialogue audio segments.

[0119] The automatic annotation module 702 is used to extract speech features from speech data and text features from text data using a pre-trained model, and to automatically annotate speech emotion and text emotion based on the speech features and text features.

[0120] The pre-training model in the embodiments of the present application is obtained by training a large amount of data, and two different pre-training models can be used to extract speech features in speech data and text features in text data respectively, and the speech features are used for speech emotion classification, and the text features are used for text emotion classification, and then speech emotion labeling and text emotion labeling are obtained.

[0121] The speech and text feature extraction module 703 is configured to extract speech feature embeddings and text feature embeddings from the speech features and the text features respectively.

[0122] By extracting speech feature embeddings from speech features and text feature embeddings from text features, complex temporal, local and global dependencies related to emotional expression in speech data and text data can be captured, thereby improving the performance and generalization ability of the model.

[0123] The emotion inconsistency feature extraction module 704 is configured to extract emotion representations from the speech features and the text features using a multi-head attention mechanism, and analyze the degree of mismatch between the emotion representations in the speech features and the text features using a cross-attention mechanism, to obtain emotion inconsistency feature embeddings.

[0124] Depression patients often exhibit special emotional expressions in their speech. For example, depression patients may convey negative emotional content in an unexpectedly calm manner, showing a high degree of inconsistency in emotional expression. These unique phenomena can serve as valuable indicators for assessing symptoms. Therefore, the embodiments of the present application use this inconsistency phenomenon to improve the accuracy of depression diagnosis.

[0125] The result prediction module 705 is configured to fuse the speech feature embeddings, the text feature embeddings and the emotion inconsistency feature embeddings, and predict the depression category according to the fused features.

[0126] By combining the emotion inconsistency feature embeddings with the identified speech feature embeddings and text feature embeddings in the speech and text fields, emotion expression inconsistency information highly related to depression can be effectively obtained.

[0127] The voice data and the text data of the target speaker are acquired from the original doctor-patient conversation voice segment, the voice features in the voice data and the text features in the text data are extracted by using a pre-trained model, and the voice emotion and the text emotion are automatically labeled according to the voice features and the text features. The automatic emotion labeling method is adopted, so that the labeling efficiency is improved, and the cost and error of manual labeling are reduced. Then, the voice feature embedding and the text feature embedding are extracted from the voice features and the text features respectively. The emotion representation is extracted from the voice features and the text features by using the multi-head attention mechanism, and the mismatch degree between the emotion representations in the voice features and the text features is analyzed by using the cross-attention mechanism, so as to obtain the emotion inconsistency feature embedding. The concept of emotion inconsistency is introduced, the difference between the emotion expressions of the depression patients in the voice and the text is compared, the emotion inconsistency information highly related to depression is effectively captured, and the reliability of depression detection can be assisted to be improved. Finally, the voice feature embedding, the text feature embedding and the emotion inconsistency feature embedding are fused, and the depression category is predicted according to the fused features, so that the emotion information in the multi-modal data is effectively integrated and analyzed, the accuracy of depression detection is increased, and the information omission or conflict in the prior art can be improved.

[0128] Based on the above, the embodiment of the present application provides a scientific, intelligent and effective automatic depression detection system for early identification and intervention of depression, especially for mild or early depression patients who are difficult to be accurately detected in traditional detection methods. The accuracy of depression detection is improved, the efficiency and user experience of the automatic depression detection system are improved, the operation process is simplified by establishing an automatic process, and the detection of depression is more convenient and efficient.

[0129] In an optional embodiment, the data preprocessing module 701 comprises:

[0130] The first data preprocessing submodule separates and identifies the speaker of the original doctor-patient conversation voice segment by using speaker log technology, and obtains the voice data of the target speaker.

[0131] The second data preprocessing submodule identifies the voice data of the target speaker by using automatic speech recognition technology, and obtains the text data of the target speaker.

[0132] In an optional embodiment, the pre-trained model comprises a HuBERT model and a RoBERTa model, and the automatic labeling module 702 comprises:

[0133] The first automatic labeling submodule is configured to extract the voice features in the voice data by using the HuBERT model, and perform voice emotion classification by using the Transformer model.

[0134] The second automatic labeling sub-module is configured to extract text features in the text data by using a RoBERTa model, and perform text sentiment classification by using a Transformer model.

[0135] In an optional embodiment, the voice and text feature extraction module 703 comprises:

[0136] The first voice and text feature extraction sub-module is configured to input the voice features and the text features into multiple Transformer blocks respectively, and obtain voice feature embeddings and text feature embeddings after average pooling layer processing.

[0137] In an optional embodiment, the sentiment inconsistency feature extraction module 704 comprises:

[0138] The first sentiment inconsistency feature extraction sub-module is configured to input the voice features and the text features into multiple Transformer blocks respectively, and obtain voice sentiment features and text sentiment features.

[0139] The second sentiment inconsistency feature extraction sub-module is configured to obtain sentiment inconsistency feature embeddings by using a cross-attention layer after linear transformation of the voice sentiment features and the text sentiment features.

[0140] The Transformer blocks use a multi-head attention mechanism to extract the voice sentiment features and the text sentiment features, and the cross-attention layer uses a cross-attention mechanism to obtain the sentiment inconsistency feature embeddings.

[0141] In an optional embodiment, the second sentiment inconsistency feature extraction sub-module comprises:

[0142] The first sentiment inconsistency feature extraction unit is configured to obtain first and second representation vectors of the inconsistency degree between the voice sentiment expression and the text sentiment expression by using a cross-attention layer after linear transformation of the voice sentiment features and the text sentiment features.

[0143] The second sentiment inconsistency feature extraction unit is configured to obtain a splicing vector by splicing the voice sentiment features, the text sentiment features, the first representation vector, and the second representation vector.

[0144] The third sentiment inconsistency feature extraction unit is configured to input the splicing vector into a full connection layer to determine whether the sentiment expressions are consistent.

[0145] The fourth sentiment inconsistency feature extraction unit is configured to use hidden outputs in the cross-attention layer as the sentiment inconsistency feature embeddings.

[0146] The depression detection system based on sentiment expression provided in the embodiments of the present application can realize Figures 1 to 6The various processes implemented in the method embodiments of the application will not be described again here to avoid repetition.

[0147] The depression detection system based on emotional expression in the embodiments of the application can execute the depression detection method based on emotional expression provided by the embodiments of the application, and the implementation principles are similar. The actions performed by each module and unit in the depression detection system based on emotional expression in the embodiments of the application correspond to the steps in the depression detection method based on emotional expression in the embodiments of the application. For the detailed functions of each module of the depression detection system based on emotional expression, refer to the description of the corresponding depression detection method based on emotional expression provided in the foregoing embodiments of the application, which will not be described again here.

[0148] Based on the same principles as the methods shown in the embodiments of the application, the embodiments of the application also provide an electronic device, which can include but is not limited to a processor and a memory. The memory is configured to store a computer program. The processor is configured to execute the depression detection method based on emotional expression shown in any optional embodiment of the application by invoking the computer program. Compared with the prior art, the depression detection method based on emotional expression provided by the embodiments of the application can detect depression based on emotional expression.

[0149] In an optional embodiment, an electronic device is also provided, which can be, for example, a server, and includes a processor and a memory. Figure 8 As shown in the foregoing embodiments of the application, Figure 8 The electronic device 800 shown in the foregoing embodiments of the application can be a server, which includes a processor 801 and a memory 803. The processor 801 and the memory 803 are connected, for example, through a bus 802. Optionally, the electronic device 800 can also include a transceiver 804. It should be noted that the transceiver 804 is not limited to one in actual applications, and the structure of the electronic device 800 does not constitute a limitation on the embodiments of the application.

[0150] The processor 801 can be a CPU (Central Processing Unit, central processing unit), a general-purpose processor, a DSP (Digital Signal Processor, digital signal processor), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure of the application. The processor 801 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0151] The bus 802 can include a path that transmits information between the above-described components. The bus 802 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 802 can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 8 In the drawing, only one thick line is shown, but this does not mean that there is only one bus or only one type of bus.

[0152] The memory 803 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0153] The memory 803 is used to store application program codes for implementing the scheme of the present application, and is controlled by the processor 801 to perform. The processor 801 is used to execute the application program codes stored in the memory 803 to realize the content shown in the foregoing method embodiments.

[0154] The electronic device includes, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a car terminal (for example, a car navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 8 The electronic device shown is only an example, and should not bring any limitation to the function and use range of the embodiments of the present application.

[0155] The server provided in the application can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like, but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the application.

[0156] The embodiment of the application provides a computer readable storage medium, which stores a computer program. When the computer program is run on a computer, the computer can execute the corresponding content in the foregoing method embodiment.

[0157] It should be understood that, although each step in the flowchart of the accompanying drawings is displayed in sequence according to the indication of the arrow, these steps are not necessarily executed in sequence according to the indication of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0158] It should be noted that the computer readable storage medium in the above embodiments of the present application can also be a computer readable signal medium or a combination of a computer readable storage medium and a computer readable signal medium. The computer readable storage medium may, for example, be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In this application, the computer readable signal medium can include a data signal propagated in a baseband or as a carrier wave in a propagated data signal, which bears computer readable program code. Such propagated data signal can take many forms, including but not limited to, an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that can send, propagate or transfer a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, an optical fiber, an RF (radio frequency) or the like, or any suitable combination of the above.

[0159] The computer readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and be not assembled into the electronic device.

[0160] The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0161] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the depression detection method and system based on emotional expression provided in the various optional implementation manners described above.

[0162] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0163] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0164] The modules involved in the embodiments of the present application can be implemented in software or hardware. In some cases, the names of the modules do not constitute a limitation on the modules themselves. For example, the data preprocessing module can also be described as a "data preprocessing module for obtaining voice data and text data of a target speaker from an original doctor-patient conversation voice segment".

[0165] The above description is merely that of preferred embodiments of the present application and of the principles thereof. It is to be understood that the disclosed scope of the present application is not limited to the specific combinations of technical features described above, and also encompasses other technical solutions formed by any combination of the above technical features or equivalent features thereof without departing from the above disclosed concept. For example, the above technical features can be replaced with technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

Claims

1. A depression detection method based on emotional expression, characterized in that, The method comprises: obtaining voice data and text data of a target speaker from an original doctor-patient conversation voice segment; extracting voice features in the voice data and text features in the text data using a pre-trained model, and automatically labeling voice emotion and text emotion according to the voice features and the text features; extracting voice feature embeddings and text feature embeddings from the voice features and the text features respectively; extracting emotion representations from the voice features and the text features using a multi-head attention mechanism, and analyzing the degree of mismatch between emotion representations in the voice features and the text features using a cross-attention mechanism to obtain emotion inconsistency feature embeddings; fusing the voice feature embeddings, the text feature embeddings and the emotion inconsistency feature embeddings, and predicting depression categories according to the fused features. 2.The depression detection method based on emotional expression according to claim 1, wherein, The voice data and text data of the target speaker are obtained from the original doctor-patient conversation voice segment, comprising: using speaker log technology to separate and identify the speaker of the original doctor-patient conversation voice segment to obtain the voice data of the target speaker; using automatic speech recognition technology to recognize the voice data of the target speaker to obtain the text data of the target speaker. 3.The depression detection method based on emotional expression according to claim 1, wherein, The pre-trained model comprises a HuBERT model and a RoBERTa model; The voice features in the voice data and the text features in the text data are extracted using a pre-trained model, and voice emotion and text emotion are automatically labeled according to the voice features and the text features, comprising: extracting voice features in the voice data using a HuBERT model, and classifying voice emotion using a Transformer model; extracting text features in the text data using a RoBERTa model, and classifying text emotion using a Transformer model. 4.The depression detection method based on emotional expression according to claim 1, wherein, The voice feature embeddings and the text feature embeddings are extracted from the voice features and the text features respectively, comprising: inputting the voice features and the text features into multiple Transformer blocks respectively, and obtaining the voice feature embeddings and the text feature embeddings after average pooling layer processing. 5.The depression detection method based on emotional expression according to claim 1, wherein, The emotion representations are extracted from the voice features and the text features using a multi-head attention mechanism, and the degree of mismatch between emotion representations in the voice features and the text features is analyzed using a cross-attention mechanism to obtain emotion inconsistency feature embeddings, comprising: inputting the voice features and the text features into multiple Transformer blocks respectively to obtain voice emotion features and text emotion features; after linear transformation of the voice emotion features and the text emotion features, the emotion inconsistency feature embeddings are obtained through a cross-attention layer; wherein the Transformer blocks use a multi-head attention mechanism to extract voice emotion features and text emotion features; the cross-attention layer uses a cross-attention mechanism to obtain the emotion inconsistency feature embeddings. 6.The depression detection method based on emotional expression according to claim 5, characterized in that, The voice emotion feature and the text emotion feature are linearly changed, and then the emotion inconsistency feature embedding is obtained through a cross-attention layer. The voice emotion feature and the text emotion feature are linearly changed, and then a first representation vector and a second representation vector of the inconsistency degree between voice emotion expression and text emotion expression are obtained through a cross-attention layer. The voice emotion feature, the text emotion feature, the first representation vector and the second representation vector are spliced to obtain a spliced vector. The spliced vector is input into a full connection layer to determine whether the emotion expression is consistent. The hidden output in the cross-attention layer is taken as the emotion inconsistency feature embedding. 7.A depression detection system based on emotional expression, characterized in that, The system comprises: A data preprocessing module configured to obtain voice data and text data of a target speaker from an original doctor-patient dialogue voice segment; An automatic labeling module configured to extract voice features from the voice data and text features from the text data by using a pre-trained model, and automatically label voice emotion and text emotion according to the voice features and the text features; A voice and text feature extraction module configured to extract voice feature embedding and text feature embedding from the voice features and the text features, respectively; An emotion inconsistency feature extraction module configured to extract emotion representation from the voice features and the text features by using a multi-head attention mechanism, and analyze the mismatching degree between emotion representations in the voice features and the text features by using a cross-attention mechanism, to obtain emotion inconsistency feature embedding; A result prediction module configured to fuse the voice feature embedding, the text feature embedding and the emotion inconsistency feature embedding, and predict a depression class according to the fused features.

8. The depression detection system based on emotional expression according to claim 7, characterized in that, The data preprocessing module comprises: A first data preprocessing submodule configured to separate and identify speakers of the original doctor-patient dialogue voice segment by using speaker log technology, to obtain voice data of a target speaker; A second data preprocessing submodule configured to recognize the voice data of the target speaker by using automatic speech recognition technology, to obtain text data of the target speaker.

9. An electronic device, comprising: A computer readable storage medium having stored thereon a computer program, the computer program being executable on a processor to implement the method of any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium has stored thereon a computer program, the computer program being executable on a processor to implement the method of any one of claims 1-6.

Images