Multi-modal depression screening evaluation method and system based on voice semantic analysis

Abstract

The application provides a multi-modal depression screening evaluation method and system based on voice semantic analysis, and the method comprises the following steps: responding to the dialogue input and dialogue output between a user and a dialogue interface; collecting, managing and analyzing the dialogue information between the user and the dialogue interface by using a dialogue management module; further performing voice recognition, semantic recognition and feature extraction on the collected dialogue information by using a pre-trained single-modal emotion recognition model; and performing multi-modal fusion on the extracted features to obtain evaluation indexes to comprehensively and objectively evaluate the depression degree. The application can well make up for the problems of the conventional depression screening test, such as strong subjectivity, strong concealability, many and complex test questions and the like. In addition, the application has low cost, is easy to popularize, can efficiently and rapidly identify the depression status of the to-be-tested personnel in a large quantity, and can be used as an effective auxiliary means for doctors to diagnose depression.

Description

Technical Field

[0001] This invention belongs to the field of depression screening and assessment technology, specifically involving a multimodal depression screening and assessment method and system based on speech semantic analysis. Background Technology

[0002] Depression is a mental illness characterized by depressed mood and loss of interest as its core symptoms. With its high prevalence, high relapse rate, high disability rate, and high mortality rate, it is the leading cause of death among mental illnesses. Early diagnosis and treatment are crucial to mitigating the harm of depression, but currently, effective identification technologies based on objective indicators are lacking. Clinical observations and studies have found that the speech behaviors of depressed patients are characterized by slowness, monotony, low pitch, and pauses, which are unusual for ordinary people. Therefore, depression recognition technology based on speech signals has become a new research hotspot due to its advantages of being inexpensive, easy to collect, and non-contact.

[0003] Against this backdrop, with the widespread application of artificial intelligence (AI) technology, researchers are attempting to develop AI-based detection methods for depression to assist medical professionals. Utilizing AI to assist doctors in patient screening and identification, thereby reducing their workload, has significant practical implications. Particularly in hospitals with limited psychiatric specialists, using AI to assist in the diagnosis of depression can improve the recognition rate and enable patients to receive intervention and treatment earlier. Currently, many scholars have conducted research on depression detection based on voice and video, but the accuracy of depression detection in real-world environments still needs improvement.

[0004] Among existing AI-based methods for detecting depression, those based on facial expressions and speech are predominant. In facial expression-based feature extraction, AUSs and Landmarks have been proven effective, but low-dimensional handcrafted features still cannot represent the entire facial information, resulting in significant information loss. In audio-based feature extraction, although there are various methods for acoustic feature extraction, there is a lack of simple, automatically extractable feature sets with strong generalization ability that can eliminate differences caused by variations in features or different implementations of the same feature. For speech emotion recognition, research from Massachusetts General Hospital can identify 12 emotions with an accuracy of 70%. For speech MCI recognition, leading technologies both domestically and internationally can only achieve around 90% accuracy on small samples, requiring approximately 20 minutes.

[0005] In addition, methods include those based on questionnaires, social media, and eye-tracking or brain imaging devices. For example, Kohrt et al. explored the effectiveness of a questionnaire based on the PhQ-9 diagnostic criteria for depression in detecting depression; Islam et al. extracted dictionary features from text posted by users on social media and used a decision tree model for depression detection; Ay et al. proposed using Long Short-Term Memory (LSTM) networks and Convolutional Neural Networks (CNNs) to process EEG data for depression detection. However, questionnaire-based depression detection methods often suffer from limited feedback and less objective and accurate results; social media-based methods require users to have sufficient posts and behaviors on social media, making them unsuitable for new users and those with sparse activity; and eye-tracking and EEG-based methods are expensive, leading to high detection costs. Furthermore, these methods involve relatively limited modalities, resulting in unsatisfactory accuracy in depression detection.

[0006] Human psychology is complex, and physiological responses vary from person to person. Using a single modality of physiological signals is insufficiently comprehensive and accurate, leading to decreased classification and prediction capabilities in different situations. The solution is to integrate multiple physiological signals and construct a multimodal physiological feature fusion strategy, leveraging the complementarity between different types of physiological signals to ensure better reliability and accuracy.

[0007] In view of this, a multimodal depression screening and assessment method and system based on speech semantic analysis is proposed, which is of great significance. Summary of the Invention

[0008] To address the issues of limited modalities in existing depression detection methods and the need to improve detection accuracy, this invention provides a multimodal depression screening and assessment method and system based on speech semantic analysis to resolve the aforementioned technical deficiencies.

[0009] In a first aspect, this invention proposes a multimodal depression screening and assessment method based on speech semantic analysis, which includes the following steps:

[0010] Responds to dialogue input and output between the user and the dialogue interface;

[0011] The dialogue management module is used to collect, manage, and analyze dialogue information between users and the dialogue interface.

[0012] Furthermore, a pre-trained single-modal emotion recognition model is used to perform speech and semantic recognition and feature extraction on the collected dialogue information; and

[0013] The extracted features are fused using multimodal methods to obtain evaluation indicators for a comprehensive and objective assessment of the degree of depression.

[0014] Preferably, the dialogue interface uses CrazyTalk, Character Creator, and iClone to build virtual characters in Reallusion4. The constructed virtual characters include facial features, lip shapes for different voices, skin textures, and character animations, and use Microsoft Azure5 to apply emotional text-to-speech synthesis technology to provide human-like speech with emotional expression.

[0015] More preferably, the dialogue management module includes a depression assessment script and a voice dialogue system. The depression assessment script refers to commonly used scales for clinical assessment of depression, including four dimensions: depressive symptoms, manic symptoms, anxiety symptoms, and family parenting style. It has eight factors: depressive factor, somatic factor, agitation factor, emotional instability factor, elevated mood factor, anxiety factor, family relationship care factor, and family autonomy factor. The voice dialogue system is implemented based on the RASA6 framework and includes a single-modal emotion recognition model and dialogue management strategies.

[0016] More preferably, the single-modal emotion recognition model includes speech recognition and semantic recognition. The speech recognition utilizes the emotion knowledge-enhanced pre-trained model SKEP, and uses sentence-level emotion classification to predict the probability of positive and negative emotions. Predicted probabilities below a threshold are classified as neutral emotions.

[0017] The speech recognition model is trained by training different classifiers and five-fold cross-validation. The valence-arousal model maps various emotion labels from different datasets to positive, negative and neutral emotions. The emotion labels are divided into three segments [-3, -1], [-1, 1] and [1, 3], which correspond to negative, neutral and positive emotions, respectively.

[0018] More preferably, the dialogue management strategy uses a 3-pass algorithm based on emotion perception to decide whether to ask for further information or proceed to the next question, specifically including:

[0019] In the first pass, the "yes" or "no" intent is determined based on the question, and the user intent is identified by the intent classifier built using the Rasa framework;

[0020] Next, the emotion consistency detection block confirms whether the identified intent is consistent with the unimodal emotion recognition results from text and audio.

[0021] If no intent is detected, the dialogue system will continue with the other questions in the series.

[0022] If the probability of intent recognition does not exceed the threshold or the recognized intent is inconsistent with the emotion recognition, the process proceeds to the second round.

[0023] In the second pass, semantic sentiment recognition is independent of the sentiment consistency detection block. If the probability of semantic sentiment recognition does not exceed the threshold or the sentiment detected in the text is different from other single-modal sentiment recognition, the third pass is executed.

[0024] In the third and final round, the next question is decided by a majority vote based on sentiment recognition results, which do not include the text.

[0025] More preferably, the feature extraction includes semantic feature extraction and speech feature extraction. The semantic feature extraction uses a pre-trained language model BERT. The model includes two tasks: masked language modeling and next sentence prediction. The pre-labeled text is input into the pre-trained BERT model, and the output of the last layer is selected to extract a vector of length 768 as the text feature vector.

[0026] The speech feature extraction includes using the Librosa8 software package to extract five spectral features from the audio file: Mel spectrogram, Mel cepstral coefficients (MFCC), spectral contrast, chromaticity map, and Tonnetz centroid feature. These spectral features are then used to form an audio feature vector of length 193.

[0027] More preferably, the multimodal fusion includes extracting text and audio modal features when answering each question, and using a decision layer and feature layer fusion method to comprehensively and objectively evaluate the degree of depression;

[0028] The decision layer fusion uses different machine learning algorithms to build multiple single-peak classifiers for text and audio, trains the model, selects the single-peak classifier with the best performance, and uses these classifiers to determine the final depression level prediction through majority voting.

[0029] The feature layer fusion utilizes a deep neural network to integrate all information from both text and audio modal features. Given text and audio modal features as input, a deep neural network with a softmax layer generates probabilities of different depression levels. All feature vectors from both modalities are directly concatenated as input and fed into a deep neural network including two hidden layers.

[0030] More preferably, the evaluation metrics include a weighted average of precision, recall, and F1 score to assess multimodal depression levels, including five levels: healthy, mild, moderate or severe depression, and bipolar disorder. The calculation formula is as follows:

[0031]

[0032]

[0033]

[0034]

[0035] Where N is the number of categories, and weight i is the ratio of category i to the total number of samples, which is equal to:

[0036]

[0037] Precision i, recall i, and F1 score i are calculated as follows:

[0038]

[0039]

[0040]

[0041] Secondly, embodiments of the present invention also provide a multimodal depression screening and assessment system based on speech semantic analysis, the system comprising:

[0042] The dialog interface module is used for dialogue input and output with the user.

[0043] The dialogue management module is used to collect, manage, and analyze dialogue information between users and the dialogue interface;

[0044] A single-modal emotion recognition model module is used to perform speech recognition and semantic recognition on the collected dialogue information and extract features.

[0045] The multimodal fusion module is used to fuse extracted features across multiple modalities to provide a comprehensive and objective evaluation of depression levels.

[0046] The evaluation metrics module is used to assess multimodal depression levels using a weighted average of precision, recall, and F1 score.

[0047] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0048] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0049] Compared with the prior art, the beneficial results of the present invention are as follows:

[0050] (1) This invention proactively guides users through voice dialogue and uses emotion perception to alter the dialogue content. During the dialogue, features are extracted from text and audio for multimodal depression level assessment. A feature-level fusion framework is used to integrate the two modalities, and a deep neural network is used to classify different levels of depression, including healthy, mild, moderate, or severe depression, as well as bipolar disorder. This invention effectively overcomes the problems of conventional depression screening tests, such as strong subjectivity, high susceptibility to concealment, and numerous and complex test questions. In addition, it is low-cost, easy to promote, and can identify the depressive status of a large number of test subjects efficiently and quickly, serving as an effective auxiliary means for doctors to diagnose depression.

[0051] (2) This invention develops a dialogue script based on a depression screening scale, integrates speech semantic analysis and artificial intelligence multimodal fusion technology, and creates a mobile software for psychological counseling dialogue using artificial intelligence. It proposes an artificial intelligence solution for multimodal depression screening; breaks the limitations of traditional psychological counseling in terms of location and time, and provides a judgment of depressive tendency through multimodal recognition.

[0052] (3) The multimodal depression screening and assessment system based on speech semantic analysis disclosed in this invention collected clinical data from 168 cases at the Shanghai Mental Health Center. The accuracy of the depression diagnosis / assessment model reached 90.26%, effectively identifying mild, moderate and severe depression, bipolar disorder and healthy individuals. Attached Figure Description

[0053] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0054] Figure 1 This is an exemplary device architecture diagram in which an embodiment of the present invention can be applied;

[0055] Figure 2 This is a flowchart illustrating the multimodal depression screening and assessment method based on speech semantic analysis, as an embodiment of the present invention.

[0056] Figure 3 This is a schematic diagram of the overall process of the multimodal depression screening and assessment method based on speech semantic analysis according to an embodiment of the present invention;

[0057] Figure 4This is a schematic diagram illustrating the dialogue script design based on depression assessment in the multimodal depression screening and assessment method based on speech semantic analysis, as described in an embodiment of the present invention.

[0058] Figure 5 This is a schematic diagram illustrating the feature extraction of speech semantics in the multimodal depression screening and assessment method based on speech semantic analysis according to an embodiment of the present invention;

[0059] Figure 6 This is a flowchart illustrating the 3-pass algorithm in the multimodal depression screening and assessment method based on speech semantic analysis, as described in an embodiment of the present invention.

[0060] Figures 7(a) and 7(b) are schematic diagrams of the architecture of the decision-level and feature-level fusion framework for multimodal depression assessment in the speech semantic analysis-based multimodal depression screening and assessment method of the present invention, respectively.

[0061] Figure 8 This is a schematic diagram of the architecture of a multimodal depression screening and assessment method based on speech semantic analysis, according to an embodiment of the present invention.

[0062] Figure 9 This is a schematic diagram of the structure of a computer device suitable for implementing electronic devices according to embodiments of the present invention. Detailed Implementation

[0063] In the following detailed description, reference is made to the accompanying drawings, which form part of the detailed description and are illustrated by specific illustrative embodiments in which the invention may be practiced. In this regard, directional terms such as “top,” “bottom,” “left,” “right,” “up,” “down,” etc., are used with reference to the orientation of the described figures. Because components of the embodiments can be positioned in several different orientations, directional terms are used for illustrative purposes and are by no means limiting. It should be understood that other embodiments may be utilized or logical changes may be made without departing from the scope of the invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the invention is defined by the appended claims.

[0064] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0065] Figure 1 An exemplary system architecture 100 for processing information, or for processing information, to which embodiments of the present invention can be applied, is shown.

[0066] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0067] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0068] Terminal devices 101, 102, and 103 can be various electronic devices with communication functions, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0069] Server 105 can be a server that provides various services, such as a background information processing server that processes verification request information sent by terminal devices 101, 102, and 103. The background information processing server can analyze and process the received verification request information and obtain processing results (such as verification success information used to indicate that the verification request is a valid request).

[0070] It should be noted that the information processing method provided in the embodiments of the present invention is generally executed by server 105, and correspondingly, the device for processing information is generally disposed in server 105. Furthermore, the information sending method provided in the embodiments of the present invention is generally executed by terminal devices 101, 102, and 103, and correspondingly, the device for sending information is generally disposed in terminal devices 101, 102, and 103.

[0071] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (for example, used to provide distributed services), or as a single software program or multiple software modules; no specific limitations are made here.

[0072] This invention develops a dialogue script based on a depression screening scale, integrates speech and semantic analysis with artificial intelligence multimodal fusion technology, and creates a mobile software for psychological counseling dialogue using artificial intelligence. It proposes an artificial intelligence solution for multimodal depression screening; it breaks through the limitations of traditional psychological counseling in terms of location and time, and provides a judgment of depressive tendencies through multimodal recognition.

[0073] Figure 2 An embodiment of the present invention discloses a multimodal depression screening and assessment method based on speech semantic analysis, such as... Figure 2 and Figure 3 As shown, the method includes the following steps:

[0074] S1, responds to dialogue input and output between the user and the dialog interface;

[0075] Specifically, in this embodiment, the dialogue interface uses CrazyTalk, Character Creator, and iClone to build virtual characters in Reallusion4, including facial features, lip shapes for different voices, skin texture, and character animation. Emotional text-to-speech synthesis technology is applied using Microsoft Azure5 to provide human-like speech with emotional expression.

[0076] S2. Use the dialogue management module to collect, manage, and analyze dialogue information between the user and the dialogue interface;

[0077] Specifically, dialogue management includes depression assessment scripts and voice dialogue systems.

[0078] Reference Figure 4 The depression assessment script partially references commonly used clinical scales for assessing depression, selecting four dimensions—depressive symptoms, manic symptoms, anxiety symptoms, and family parenting styles—to design the dialogue script questions. Each dimension includes multiple factors:

[0079] a. Depressive symptoms dimension: depressive factor, somatic factor;

[0080] b. Manic symptoms dimensions: excitatory factors, mood instability factors, and elevated mood factors;

[0081] c. Anxiety symptom dimension: Anxiety factors;

[0082] d. Family parenting style dimension: family relationship care factor, family autonomy factor.

[0083] The voice dialogue system is implemented using RASA6, a framework specifically designed for building dialogue assistants and chatbots. The system includes a single-modal emotion recognition model and dialogue management strategies.

[0084] S3. Further utilize a pre-trained single-modal emotion recognition model to perform speech recognition and semantic recognition on the collected dialogue information and extract features; and

[0085] Specifically, the single-modal emotion recognition model includes speech recognition and semantic recognition.

[0086] The semantic recognition component utilizes a sentiment knowledge-enhanced pre-trained (SKEP) model, a pre-training algorithm that leverages sentiment knowledge for reinforcement. This model learns to understand sentiment semantics spontaneously from mined sentiment knowledge through unsupervised learning methods. Sentence-level sentiment classification is used to predict the probabilities of positive and negative emotions; prediction probabilities below a threshold are classified as neutral.

[0087] For speech recognition, refer to Figure 5 The details of the input features are described. In this embodiment, we trained different classifiers and validated the accuracy on public datasets, including CMU-MOSEI, IEMOCAP, RAVDESS, EmoDB, MAHNOB-HCI, and SEED-IV. All models were trained using five-fold cross-validation, and a valence-arousal model was used to uniformly map various sentiment labels from different datasets to positive, negative, and neutral sentiments. Specifically, the sentiment labels for CMU-MOSEI range from -3 to 3. We divided the labels into three segments: [-3, -1], [-1, 1], and (1, 3], corresponding to negative, neutral, and positive sentiments, respectively.

[0088] In this embodiment, refer to Figure 6 The dialogue management strategy uses a 3-pass algorithm based on emotion perception to decide whether to ask for further information or proceed to the next question. In the first pass, the "yes" or "no" intent is determined based on the question. For example, "Have you been emotionally unstable lately?" or "Do you sometimes lose your temper?" The intent classifier, built using the Rasa framework, identifies the user's intent.

[0089] Next, the emotion consistency detection block confirms whether the identified intent is consistent with the unimodal emotion recognition results from the text and audio. In the previous example, the "yes" intent was associated with negative emotions, so another question is asked: Does this unstable emotion sometimes manifest as unhappiness and sometimes as sudden agitation and a desire to lash out? The "no" intent is associated with positive emotions. If a "no" intent is detected, the dialogue system continues with the other questions in the series. If the probability of intent recognition does not exceed a threshold or the identified intent is inconsistent with the emotion recognition, the process proceeds to a second iteration.

[0090] In the second pass, semantic sentiment recognition is performed independently of the sentiment consistency detection block. A third pass is executed once the probability of semantic sentiment recognition does not exceed a threshold or the detected sentiment in the text differs from other unimodal sentiment recognitions. In the third and final pass, the next question is determined by a majority vote based on the sentiment recognition results, excluding the text itself.

[0091] Furthermore, multimodal depression assessment includes feature extraction, multimodal fusion, and evaluation indicators.

[0092] Specifically, feature extraction includes semantic feature extraction and speech feature extraction. Semantic feature extraction uses the Bidirectional Encoder Representations from Transformers (BERT) model, a powerful pre-trained language model for general purposes that has achieved state-of-the-art results in numerous natural language processing tasks. We imported a pre-trained BERT model from Hugging Face7, which had been trained on a large corpus and included two tasks: masked language modeling and next-sentence prediction. The BERT-based model consists of a 12-layer transform encoder with 12 bidirectional self-attention heads and contains 110M parameters. We input tokenized text into the pre-trained BERT model and selected the output of the last layer, a vector of length 768, as the text feature representation.

[0093] 768 is a parameter set in the BERT model (e.g., bert-base-chinese: the encoder has 12 hidden layers, outputs a 768-dimensional tensor, 12 self-attention heads, and a total of 110M parameters, obtained by training on simplified and traditional Chinese text).

[0094] Speech feature extraction used the Librosa8 software package to extract five spectral features from the audio file: Mel spectrogram, Mel cepstral coefficients (MFCC), spectral contrast, chromaticity map, and Tonnetz features. The Mel spectrogram is the original scaled spectrogram converted to Mel scale, which represents the characteristics of the audio signal. Mel cepstral coefficients reflect the human ear's perception of different frequencies and have wide applications in speaker recognition and speech recognition. The octave-based spectral contrast feature indicates the relative spectral distribution that estimates the difference between spectral peaks and troughs in each subband. Chromaticity map features were computed from the audio, where the complete spectrum was projected onto 12 boxes representing 12 different semitones of a musical octave. Tonnetz mapped the 12 interval chromaticity vectors onto a 6-dimensional basis capable of detecting harmonic variations. Finally, all features formed an audio feature vector of length 193.

[0095] S4. Multimodal fusion of the extracted features is performed to obtain evaluation indicators for a comprehensive and objective evaluation of the degree of depression.

[0096] Specifically, referring to Figures 7(a) and 7(b), multimodal fusion refers to extracting two modal features—text and audio—when answering each question during the user-system dialogue. Based on this, a decision-level and feature-level fusion method is used to achieve a comprehensive and objective evaluation of depression levels. The model is implemented using the Python machine learning libraries scikit-learn, Tensorflow, and Keras.

[0097] For decision-level fusion, multiple unimodal classifiers for text and audio are first constructed using different machine learning algorithms, including K-Nearest Neighbors (KNN), Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), Multilayer Perceptron (MLP), Adaptive Boosting (AdaBoost), and Gradient Boosting (GB). After model training, the best-performing unimodal classifier is selected, and these classifiers are used to determine the final depression level prediction through majority voting.

[0098] For feature-level fusion, a deep neural network is used to integrate all information from two modal features. Given two modal features as input, a deep neural network with a softmax layer generates probabilities of different depression levels. All feature vectors from both modalities are directly concatenated as input and fed into the deep neural network, which includes two hidden layers. The first hidden layer is followed by a dropout layer with a probability of 0.2, containing 512 neurons, and L2 regularization is performed. The second hidden layer has 256 neurons and is also followed by a dropout layer with a probability of 0.2. For the deep neural network with feature-level fusion, we train it for 100 epochs and choose Adam as the optimizer with a learning rate of 0.001.

[0099] Furthermore, for evaluation metrics, multimodal depression level assessment is a multi-class cognitive task. For multi-class classification, precision and a weighted average of precision, recall, and F1 score are used to evaluate model performance and validate generalization ability, including five levels: healthy, mild, moderate or severe depression, and bipolar disorder. The formula is as follows:

[0100]

[0101]

[0102]

[0103]

[0104] Where N is the number of categories, and weight i is the ratio of category i to the total number of samples, which is equal to:

[0105]

[0106] Precision i, recall i, and F1 score i are calculated as follows:

[0107]

[0108]

[0109]

[0110] This invention proactively guides users through voice dialogue and utilizes emotion perception to modify the dialogue content. During the dialogue, features are extracted from text and audio for multimodal depression level assessment. A feature-level fusion framework integrates the two modalities, and a deep neural network is used to classify different levels of depression, including healthy, mild, moderate, or severe depression, as well as bipolar disorder. This invention effectively overcomes the problems of conventional depression screening tests, such as strong subjectivity, high susceptibility to masking, and numerous and complex test items. Furthermore, it is low-cost, easy to promote, and can identify the depressive status of a large number of individuals efficiently and rapidly, serving as an effective auxiliary tool for doctors in diagnosing depression.

[0111] The multimodal depression screening and assessment system based on speech and semantic analysis disclosed in this invention collected clinical data from 168 cases at the Shanghai Mental Health Center. The accuracy of the depression diagnosis / assessment model reached 90.26%, effectively identifying mild, moderate and severe depression, bipolar disorder and healthy individuals.

[0112] Secondly, embodiments of the present invention also disclose a multimodal depression screening and assessment system based on speech semantic analysis, such as... Figure 8 As shown, the system includes: a dialogue interface module 81, a dialogue management module 82, a single-modal emotion recognition model module 83, a multimodal fusion module 84, and an evaluation index module 85.

[0113] In one specific embodiment, the dialogue interface module 81 is used for dialogue input and output with the user; the dialogue management module 82 is used for collecting, managing and analyzing dialogue information between the user and the dialogue interface; the single-modal emotion recognition model module 83 is used for speech recognition and semantic recognition and feature extraction of the collected dialogue information; the multimodal fusion module 84 is used for comprehensive and objective evaluation of the degree of depression; and the evaluation index module 85 is used for evaluating the multimodal depression level using a weighted average of precision, recall and F1 score.

[0114] The following is for reference. Figure 9 It illustrates an electronic device suitable for implementing embodiments of the present invention (e.g., Figure 1 The diagram shows the structure of a computer device 900 (a server or terminal device). Figure 9The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0115] like Figure 9 As shown, the computer device 900 includes a central processing unit (CPU) 901 and a graphics processing unit (GPU) 902, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 903 or programs loaded from storage section 909 into random access memory (RAM) 906. Various programs and data required for the operation of device 900 are also stored in RAM 904. The CPU 901, GPU 902, ROM 903, and RAM 904 are interconnected via bus 905. An input / output (I / O) interface 906 is also connected to bus 905.

[0116] The following components are connected to I / O interface 906: an input section 907 including a keyboard, mouse, etc.; an output section 908 including an LCD, speakers, etc.; a storage section 909 including a hard disk, etc.; and a communication section 910 including a network interface card, such as a LAN card or modem. The communication section 910 performs communication processing via a network such as the Internet. A drive 911 may also be connected to I / O interface 906 as needed. A removable medium 912, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 911 as needed so that computer programs read from it can be installed into storage section 909 as required.

[0117] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 910, and / or installed from removable medium 912. When the computer program is executed by central processing unit (CPU) 901 and graphics processing unit (GPU) 902, the functions defined in the methods of this invention are performed.

[0118] It should be noted that the computer-readable medium described in this invention can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution apparatus, device, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device, apparatus, or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0119] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using dedicated hardware-based devices that perform the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0121] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be located in a processor.

[0122] In another aspect, the present invention also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: respond to dialogue input and output between the user and the dialogue interface; collect, manage, and analyze dialogue information between the user and the dialogue interface using a dialogue management module; further utilize a pre-trained single-modal emotion recognition model to perform speech recognition and semantic recognition on the collected dialogue information and extract features; and obtain evaluation indicators through multimodal fusion to comprehensively and objectively evaluate the degree of depression.

[0123] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A multimodal depression screening and assessment method based on speech semantic analysis, characterized in that, The method includes the following steps: The dialogue interface is responsive to dialogue input and output between the user and the dialogue interface, which has the ability to express emotions in order to achieve human-like interaction. A dialogue management module is used to collect, manage, and analyze dialogue information between users and the dialogue interface. This module includes a depression assessment script and a voice dialogue system. The depression assessment script references commonly used clinical scales for assessing depression, including four dimensions: depressive symptoms, manic symptoms, anxiety symptoms, and parenting style. It contains eight factors: depressive factor, somatic factor, agitation factor, emotional instability factor, elevated mood factor, anxiety factor, family relationship care factor, and family autonomy factor. The voice dialogue system is implemented based on the RASA framework and includes a single-modal emotion recognition model and dialogue management strategies. Further, a pre-trained unimodal emotion recognition model is used to perform speech recognition and semantic recognition on the collected dialogue information and to extract features. The feature extraction includes semantic feature extraction and speech feature extraction. The semantic feature extraction uses a pre-trained language model BERT, which includes two tasks: masked language modeling and next sentence prediction. The pre-labeled text is input into the pre-trained BERT model, and the output of the last layer is selected to extract a vector of length 768 as the text feature vector. The speech feature extraction includes using the Librosa8 package to extract five spectral features from the audio file: Mel spectrogram, Mel cepstral coefficients (MFCC), spectral contrast, chroma map, and Tonnetz pitch centroid feature. The spectral features are extracted to form an audio feature vector of length 193. as well as Multimodal fusion of extracted features yields evaluation metrics for a comprehensive and objective assessment of depression levels. This multimodal fusion involves extracting text and audio features from each question's answers and employing a decision-level and feature-level fusion method to comprehensively and objectively evaluate depression levels. The decision-level fusion uses different machine learning algorithms to construct multiple single-peak classifiers for text and audio, trains the model, selects the best-performing single-peak classifier, and uses these classifiers to determine the final depression level prediction through majority voting. The feature-level fusion utilizes a deep neural network to integrate all information from both text and audio modal features. Given text and audio modal features as input, a deep neural network with a softmax layer generates probabilities of different depression levels. All feature vectors from both modalities are directly concatenated as input and fed into a deep neural network with two hidden layers.

2. The multimodal depression screening and assessment method based on speech semantic analysis according to claim 1, characterized in that, The dialogue interface uses CrazyTalk, Character Creator, and iClone to build virtual characters in Reallusion4. The virtual characters include facial features, lip shapes for different voices, skin textures, and character animations. Microsoft Azure5 is used to apply emotional text-to-speech synthesis technology to provide human-like speech with emotional expression.

3. The multimodal depression screening and assessment method based on speech semantic analysis according to claim 2, characterized in that, The single-modal emotion recognition model includes speech recognition and semantic recognition. The speech recognition utilizes the emotion knowledge-enhanced pre-trained model SKEP, which uses sentence-level emotion classification to predict the probability of positive and negative emotions. Predictions below a threshold are classified as neutral emotions. The speech recognition model is trained by training different classifiers and five-fold cross-validation. The valence-arousal model maps various emotion labels from different datasets to positive, negative and neutral emotions. The emotion labels are divided into three segments [-3, -1], [-1, 1] and [1, 3], which correspond to negative, neutral and positive emotions, respectively.

4. The multimodal depression screening and assessment method based on speech semantic analysis according to claim 3, characterized in that, The dialogue management strategy uses a 3-pass algorithm based on emotion perception to decide whether to ask for further information or proceed to the next question, specifically including: In the first pass, the "yes" or "no" intent is determined based on the question, and the user intent is identified by the intent classifier built using the Rasa framework; Next, the emotion consistency detection block confirms whether the identified intent is consistent with the unimodal emotion recognition results from text and audio. If no "no" intent is detected, the dialogue system will continue with the other questions in the series. If the probability of intent recognition does not exceed the threshold or the recognized intent is inconsistent with the emotion recognition, the process proceeds to the second round. In the second pass, semantic sentiment recognition is independent of the sentiment consistency detection block. Once the probability of semantic sentiment recognition does not exceed the threshold or the sentiment detected in the text is different from other single-modal sentiment recognition, the third pass is executed. In the third and final round, the next question is decided by a majority vote based on sentiment recognition results, which do not include the text.

5. The multimodal depression screening and assessment method based on speech semantic analysis according to claim 4, characterized in that, The evaluation metrics include a weighted average of precision, recall, and F1 score to assess multimodal depression levels, calculated as follows: ； Where N is the number of categories, and weight i is the ratio of category i to the total number of samples, which is equal to: ； Precision i, recall i, and F1 score i are calculated as follows: 。 6. A multimodal depression screening and assessment system based on speech and semantic analysis, characterized in that, The system includes the evaluation method as described in any one of claims 1-5, and further includes: The dialog interface module is used for dialogue input and output with the user. The dialogue management module is used to collect, manage, and analyze dialogue information between users and the dialogue interface; A single-modal emotion recognition model module is used to perform speech recognition and semantic recognition on the collected dialogue information and extract features. The multimodal fusion module is used to fuse extracted features across multiple modalities to provide a comprehensive and objective evaluation of depression levels. The evaluation metrics module is used to assess multimodal depression levels using a weighted average of precision, recall, and F1 score.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images