Voice signal-based depression auxiliary diagnosis system and diagnosis method

Abstract

The invention discloses a depression auxiliary diagnosis system and method based on voice signals. The depression auxiliary diagnosis system comprises a voice acquisition module, a feature extraction module, a diagnosis model training module and a diagnosis result output module. The voice acquisition module is used for acquiring voice signals of a subject and constructing a depression voice contrast expansion set; the feature extraction module is used for performing feature extraction on the collected voice signals and analyzing two acoustic features of eGeMAPS and VGGish; the diagnosis model training module is used for processing and fusing eGeMAPS and VGGish features, and constructing an EGV-AttNet model on the basis of an encoder of a Transform architecture; and the diagnosis result output module is used for training and storing the EGV-AttNet model, diagnosing the new voice signal and outputting a diagnosis result. The accuracy and efficiency of depression diagnosis can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a voice signal-based depression auxiliary diagnosis system and diagnosis method. Background Art

[0002] Depression is a common and serious mental disorder that greatly affects the patient's physical and mental health and quality of life. Early and accurate diagnosis is crucial for the treatment and recovery of depression. However, traditional methods for diagnosing depression mainly rely on clinicians' subjective assessments and patients' self-reports, which have certain limitations.

[0003] On the one hand, doctors' subjective judgments may be affected by personal experience, emotional state and other factors, which may challenge the accuracy and consistency of diagnostic results. On the other hand, patients' self-reports may contain concealment, false positives, etc., and for some patients whose symptoms are not obvious or difficult to express, the reliability of self-reports is low.

[0004] With the continuous development of science and technology, diagnostic methods based on physiological signals have gradually attracted the attention of researchers. As a physiological signal that is easy to obtain and rich in information, speech signals provide a new idea for the diagnosis of depression. Studies have shown that patients with depression may have specific changes in speech rhythm, speaking speed, intonation, and sound quality. By analyzing and mining these speech features, it can provide valuable reference for the early diagnosis of depression.

[0005] However, the existing methods for diagnosing depression based on speech signals still have many shortcomings. For example, the feature extraction method is not perfect and fails to fully tap the potential information in the speech signal; the accuracy and generalization ability of the diagnostic model need to be improved; the practicality and operability of the system are not strong, etc. Therefore, developing a more efficient, accurate and practical depression auxiliary diagnosis system and method based on speech signals has important practical significance and application value. Summary of the invention

[0006] Purpose of the invention: The present invention provides a depression auxiliary diagnosis system based on speech signals, which can improve the accuracy and efficiency of depression diagnosis.

[0007] Technical solution: The speech signal-based depression auxiliary diagnosis system described in the present invention includes: a speech acquisition module, a feature extraction module, a diagnosis model training module and a diagnosis result output module; the speech acquisition module is used to acquire the subject's speech signal and construct a depression speech control extension set; the feature extraction module is used to extract features from the collected speech signals and analyze the two acoustic features of eGeMAPS and VGGish; the diagnosis model training module is used to process and fuse the eGeMAPS and VGGish features, and construct an EGV-AttNet model based on an encoder of the Transformer architecture; the diagnosis result output module trains and saves the EGV-AttNet model, diagnoses new speech signals, and outputs diagnosis results.

[0008] Accordingly, a method for auxiliary diagnosis of depression based on speech signals comprises the following steps:

[0009] Step 1: Collect the voice signals of the subjects and construct an extended set of depression voice controls;

[0010] Step 2: Extract features from the speech signal collected in step 1 and analyze the two acoustic features of eGeMAPS and VGGish;

[0011] Step 3: Process and fuse the eGeMAPS and VGGish features, and build the EGV-AttNet model based on the encoder of the Transformer architecture;

[0012] Step 4: Train and save the EGV-AttNet model, diagnose the new speech signal, and output the diagnosis result.

[0013] Furthermore, in step 1, collecting the patient's voice signal includes the following steps:

[0014] Step 11, obtain demographic data of depression patients and healthy controls, including age, gender, education, marital status, previous physical illness, and previous mental illness;

[0015] Step 12, using the nine-item Patient Health Questionnaire Depression Scale (PHQ-9) as an assessment scale for depression to obtain diagnostic scale data;

[0016] Step 13: The subject reads the specified corpus and uses a specific recording tool to collect voice in an independent laboratory with an environmental noise level below 50 dB and no electromagnetic interference. The recording tool is set to a sampling frequency of 48 kHz, a sampling depth of 16 bits, and a dual-channel setting, and the collected voice signal is saved in wav format;

[0017] Step 14: Through the screening and cleaning mechanism, further select voice samples that meet the standards to construct the Depressive Voice Control Extended Set (DVCESet), establish a self-control subset and an inter-group control subset, and mark the healthy and depressive state voices.

[0018] Furthermore, in step 2, feature extraction is performed on the speech signal collected in step 1, and the analysis of the two acoustic features of eGeMAPS and VGGish specifically includes the following steps:

[0019] Step 21: Use the eGeMAPS configuration file to run the openSMILE tool through the command line. The input command format is: SMILExtract -C <config-file> -I <input-audio-file> -O <output-feature-file>;

[0020] Step 22: Use the openSMILE tool to extract 88 eGeMAPS features including pitch, jitter, loudness, spectral features, etc., and save the results as a CSV file;

[0021] Step 23: Perform weight conversion on the original VGGish model based on the TensorFlow framework and migrate it to the PyTorch framework;

[0022] Step 24: Input the speech signal into the VGGish model and convert it into a 128-dimensional VGGish feature vector;

[0023] Step 25: Classify and evaluate the importance of eGeMAPS features through machine learning techniques, and directly use deep learning networks to process and analyze VGGish features.

[0024] Furthermore, in step 3, the eGeMAPS and VGGish features are processed and fused, and the encoder based on the Transformer architecture is used to construct the EGV-AttNet model, which specifically includes the following steps:

[0025] Step 31: Use feature importance curves to weight eGeMAPS features, implement dynamic weighting based on the relevance of the task objectives, and enhance the contribution and identifiability of features in subsequent model training;

[0026] Step 32: Use a one-dimensional convolutional layer to preprocess the VGGish features to achieve spatial encoding of primary features, and input these encoded features into a series of residual network structures;

[0027] Step 33: Input the residual block output into the ECA-Net attention mechanism model. By considering the cross-dependency between feature channels, a channel attention is implemented to dynamically adjust the response strength of the feature channel, highlight the most discriminative features, and suppress redundant or irrelevant information, thereby improving the model's ability to extract key features.

[0028] Step 34: Input the ECA-Net output into the LSTM network, so that the model can focus on important events within the duration when processing the audio stream, exclude uninfluential signal changes, improve the model's ability to capture long-term dependencies, and enhance its ability to perform robust feature representation and sequence modeling in complex audio environments;

[0029] Step 35, perform feature conversion on the eGeMAPS and VGGish features processed in the above steps and input them into the feature fusion layer;

[0030] Step 36: Input the output of the feature fusion layer into the encoder based on the Transformer architecture for further encoding, and perform classification prediction through the fully connected layer. The calculation expression of the multi-head attention mechanism in the Transformer architecture is:

[0031]

[0032] Among them, Q, K, V represent query, key, and value respectively, and dk represents the dimension of the key.

[0033] Furthermore, in step 32, the residual network structure is as follows:

[0034]

[0035] Among them: x is the residual network input, Y is the residual network output, represents the operation of the convolutional layer, and wi is the weight of the convolutional layer.

[0036] Furthermore, in step 34, the LSTM network structure is as follows:

[0037] i t =σ(w xi x t +w hi h t-1 +w ci c t-1 +b i )

[0038] f t =σ(w xf x t +w hf h t-1 +w cf c t-1 +b f )

[0039] c t =f t c t-1 +i t tan(w xc x t +w hc h t-1 +b c )

[0040] o t =σ(w xo x t +w ho h t-1 +w co c t-1 +b o )

[0041] y t =o t tanc t

[0042] Where: x t Input for the current time, h t-1 is the external state at the previous moment, c t is the memory cell state, c t-1 is the memory cell state at the previous moment, b is the offset matrix, w x For x t The weight matrix, w h h t-1 The weight matrix, w c c t-1 The weight matrix, i t is the output of the input gate, f t is the output of the forget gate, o t is the output of the output gate, y t is the output value, and σ adopts the Sigmoid function.

[0043] Furthermore, in step 4, the EGV-AttNet model is trained and saved, and the new speech signal is diagnosed and the diagnosis result is output, which includes the following steps:

[0044] Step 41, using a stratified sampling method to divide the inter-group control data set into a training set, a validation set, and a test set in a ratio of 8:1:1;

[0045] Step 42: Select the weighted cross entropy loss function as the loss function for training;

[0046] Step 43: Select Adam as the optimizer for model training.

[0047] Step 44: Introduce a learning rate scheduler using the ReduceLROnPlateau strategy. When the validation set loss stops improving, the learning rate is automatically reduced to further optimize the training process.

[0048] Step 45: Use the Optuna framework to automatically and efficiently search for the optimal hyperparameter combination;

[0049] Step 46: During the hyperparameter optimization process, pruning strategy and early stopping mechanism are adopted to reduce unnecessary consumption of computing resources and improve search efficiency.

[0050] Step 47: Based on the prediction results on the validation set, select and save the best performing model, test it on the test set, output the diagnosis results, and evaluate the generalization ability of the model;

[0051] Step 48: Use the detection index module to evaluate the diagnosis results, including accuracy, precision, recall, and F1 score, to ensure the accuracy and reliability of the diagnosis results. Considering the performance difference between the current model on the validation set and the test set, use five-fold cross validation to carefully evaluate the performance of the model;

[0052] Step 49: Use the confusion matrix as the analysis object to evaluate the performance of the model in identifying positive and negative samples on the validation set and the test set.

[0053] Compared with the prior art, the present invention has the following significant advantages: the present invention innovatively proposes a deep learning model for early diagnosis of depression, which effectively obtains multi-dimensional information in speech by simultaneously creating two feature sets, eGeMAPS and VGGish, and combining acoustic and deep audio features; in addition, the diagnostic model uses LSTM to capture the temporal characteristics of speech signals, and uses the Transformer encoder for deep learning of fused speech features, thereby improving the global learning ability of the model; finally, the weighted cross entropy loss is used on the model classifier, and the category imbalance problem is solved in combination with category weights, ensuring that the deviation is reduced in the prediction of depression. The present invention has the advantages of high diagnostic accuracy, simple operation, and non-invasiveness, and can be widely used in early screening, auxiliary diagnosis and efficacy evaluation of depression, such as schools, hospitals, clinics, psychological counseling agencies, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of the deep learning model of the present invention.

[0055] Figure 2 Schematic diagram of the system accuracy of the present invention.

[0056] Figure 3 Schematic diagram of the confusion matrix of the present invention. DETAILED DESCRIPTION

[0057] like Figure 1 As shown, a depression auxiliary diagnosis system based on speech signals includes: a speech acquisition module, a feature extraction module, a diagnosis model training module and a diagnosis result output module; the speech acquisition module is used to acquire the speech signals of the subjects and construct a depression speech control extension set; the feature extraction module is used to extract features from the collected speech signals and analyze the two acoustic features of eGeMAPS and VGGish; the diagnosis model training module is used to process and fuse the eGeMAPS and VGGish features, and construct an EGV-AttNet model based on the encoder of the Transformer architecture; the diagnosis result output module trains and saves the EGV-AttNet model, diagnoses the new speech signals, and outputs the diagnosis results.

[0058] Accordingly, a method for auxiliary diagnosis of depression based on speech signals comprises the following steps:

[0059] Step 1: Collect the subject's speech signal and construct a depression speech control extension set. The main implementation steps are as follows:

[0060] (1) Obtain demographic data of patients with depression and healthy controls, including age, gender, education level, marital status, previous physical illness, and previous mental illness.

[0061] (2) The nine-item Patient Health Questionnaire Depression Scale (PHQ-9) was used as an assessment scale for depression to obtain diagnostic scale data.

[0062] (3) The subjects read the designated corpus and collected speech using a specific recording tool in an independent laboratory with an ambient noise level below 50 dB and no electromagnetic interference. The recording tool was set to a sampling frequency of 48 kHz, a sampling depth of 16 bits, and a dual-channel setting, and the collected speech signal was saved in WAV format.

[0063] (4) Through the screening and cleaning mechanism, we further selected speech samples that met the standards to construct the Depressive Voice Control Extended Set (DVCESet), established self-control subsets and inter-group control subsets, and marked the healthy and depressive state speech.

[0064] Step 2: Extract features from the collected speech signals and analyze the two acoustic features of eGeMAPS and VGGish. The main implementation steps are as follows:

[0065] (1) Use the eGeMAPS configuration file to run the openSMILE tool through the command line. The input command format is: SMILExtract -C <config-file> -I <input-audio-file> -O <output-feature-file>.

[0066] (2) Use the openSMILE tool to extract 88 eGeMAPS features including pitch, jitter, loudness, spectral features, etc., and save the results as a CSV file.

[0067] (3) Perform weight conversion on the original VGGish model based on the TensorFlow framework and migrate it to the PyTorch framework.

[0068] (4) The speech signal is input into the VGGish model and converted into a 128-dimensional VGGish feature vector.

[0069] (5) The eGeMAPS features are classified and their importance is evaluated through machine learning techniques, and the VGGish features are directly processed and analyzed using deep learning networks.

[0070] Step 3: Process and fuse the eGeMAPS and VGGish features, and build the EGV-AttNet model based on the encoder of the Transformer architecture. The main implementation steps are as follows:

[0071] (1) The eGeMAPS features are weighted using feature importance curves to achieve dynamic weighting based on the relevance of the task objective, thereby enhancing the contribution and identifiability of the features in subsequent model training.

[0072] (2) Use a one-dimensional convolutional layer to preprocess the VGGish features to achieve spatial encoding of primary features, and input these encoded features into a series of residual network structures. The residual network structure is as follows:

[0073]

[0074] Among them: x is the residual network input, Y is the residual network output, represents the operation of the convolutional layer, and wi is the weight of the convolutional layer.

[0075] (3) The residual block output is input into the ECA-Net attention mechanism model. By considering the cross-dependency between feature channels, a channel attention is implemented to dynamically adjust the response strength of the feature channel, highlight the most discriminative features, and suppress redundant or irrelevant information, thereby improving the model's ability to extract key features.

[0076] (4) The ECA-Net output is input into the LSTM network, so that the model can focus on important events within the duration when processing the audio stream, exclude uninfluential signal changes, improve the model's ability to capture long-term dependencies, and enhance its ability to perform robust feature representation and sequence modeling in complex audio environments. The LSTM network structure is as follows:

[0077] i t =σ(w xi x t +w hi h t-1 +w ci c t-1 +b i )

[0078] f t =σ(w xf x t +w hf h t-1 +w cf c t-1 +b f )

[0079] c t =f t c t-1 +i t tan(w xc x t +w hc h t-1 +b c )

[0080] o t =σ(w xo x t +w ho h t-1 +w co c t-1 +b o )

[0081] y t =o t tanc t

[0082] Where: x t Input for the current time, h t-1 is the external state at the previous moment, c t is the memory cell state, c t-1 is the memory cell state at the previous moment, b is the offset matrix, w x For x t The weight matrix, w h h t-1 The weight matrix, w c c t-1 The weight matrix, i t is the output of the input gate, f t is the output of the forget gate, o t is the output of the output gate, y t is the output value, and σ adopts the Sigmoid function.

[0083] (5) The eGeMAPS and VGGish features processed in the above steps are converted respectively and then input into the feature fusion layer.

[0084] (6) The output of the feature fusion layer is input into the encoder based on the Transformer architecture for further encoding, and classification prediction is performed through the fully connected layer. The calculation of the multi-head attention mechanism in the Transformer architecture can be expressed as:

[0085]

[0086] Among them, Q, K, V represent query, key, and value respectively, and dk represents the dimension of the key.

[0087] Step 4: Train and save the EGV-AttNet model, diagnose the new speech signal, and output the diagnosis results. The main implementation steps are as follows:

[0088] (1) The intergroup control data set was divided into training set, validation set and test set in a ratio of 8:1:1 using the stratified sampling method.

[0089] (2) Select the weighted cross entropy loss function as the loss function for training.

[0090] (3) Select Adam as the optimizer for model training.

[0091] (4) A learning rate scheduler with the ReduceLROnPlateau strategy is introduced to automatically reduce the learning rate when the validation set loss stops improving, further optimizing the training process.

[0092] (5) Use the Optuna framework to automatically and efficiently search for the optimal hyperparameter combination.

[0093] (6) During the hyperparameter optimization process, pruning strategy and early stopping mechanism are adopted to reduce unnecessary consumption of computing resources and improve search efficiency.

[0094] (7) Based on the prediction results on the validation set, select and save the model with the best performance, test it on the test set, output the diagnosis results, and evaluate the generalization ability of the model.

[0095] (8) Use the detection index module to evaluate the diagnosis results, including accuracy, precision, recall, and F1 score, to ensure the accuracy and reliability of the diagnosis results. Considering the performance difference between the current model on the validation set and the test set, a five-fold cross validation is used to carefully evaluate the performance of the model, such as Figure 2 shown.

[0096] (9) Figure 3 As shown in the figure, the confusion matrix is ​​used as the analysis object to evaluate the performance of the model in identifying positive and negative samples on the validation set and the test set. < / input-audio-file> < / config-file> < / input-audio-file> < / config-file>

Claims

1. A depression auxiliary diagnosis system based on speech signals, characterized in that: include: Speech acquisition module, feature extraction module, diagnostic model training module and diagnostic result output module; the speech acquisition module is used to collect the subject's speech signal and build a depression speech control extension set; the feature extraction module is used to extract features from the collected speech signals and analyze the two acoustic features of eGeMAPS and VGGish; The diagnosis model training module is used to process and fuse the eGeMAPS and VGGish features, and build the EGV-AttNet model based on the Transformer architecture encoder; the diagnosis result output module trains and saves the EGV-AttNet model, diagnoses the new speech signal, and outputs the diagnosis result.

2. A diagnostic method based on the depression auxiliary diagnosis system based on speech signals as claimed in claim 1, characterized in that: The steps include: Step 1: Collect the voice signals of the subjects and construct an extended set of depression voice controls; Step 2: Extract features from the speech signal collected in step 1 and analyze the two acoustic features of eGeMAPS and VGGish; Step 3: Process and fuse the eGeMAPS and VGGish features, and build the EGV-AttNet model based on the encoder of the Transformer architecture; Step 4: Train and save the EGV-AttNet model, diagnose the new speech signal, and output the diagnosis result.

3. The method for auxiliary diagnosis of depression based on speech signals according to claim 2, characterized in that: In step 1, collecting the patient's voice signal includes the following steps: Step 11, obtain demographic data of depression patients and healthy controls; Step 12, using the nine-item Patient Health Questionnaire Depression Scale as an assessment scale for depression to obtain diagnostic scale data; Step 13: The subject reads the specified corpus and uses a specific recording tool to collect voice in an independent laboratory with an environmental noise level below 50 dB and no electromagnetic interference. The recording tool is set to a sampling frequency of 48 kHz, a sampling depth of 16 bits, and a dual-channel setting, and the collected voice signal is saved in wav format; Step 14: Through the screening and cleaning mechanism, further select speech samples that meet the standards to construct the depression speech control extended set DVCESet, establish the self-control subset and the inter-group control subset, and mark the healthy and depressed state speech.

4. The method for auxiliary diagnosis of depression based on speech signals according to claim 3, characterized in that: In step 11, demographic data include age, gender, education, marital status, previous physical illness, and previous mental illness.

5. The method for auxiliary diagnosis of depression based on speech signals according to claim 2, characterized in that: In step 2, feature extraction is performed on the speech signal collected in step 1, and the analysis of the two acoustic features of eGeMAPS and VGGish specifically includes the following steps: Step 21: Use the eGeMAPS configuration file to run the openSMILE tool through the command line. The input command format is: SMILExtract -C <config-file> -I <input-audio-file> -O <output-feature-file> ；< / output-feature-file> < / input-audio-file> < / config-file> Step 22: Use the openSMILE tool to extract 88 eGeMAPS features including pitch, jitter, loudness, spectral features, etc., and save the results as a CSV file; Step 23: Perform weight conversion on the original VGGish model based on the TensorFlow framework and migrate it to the PyTorch framework; Step 24: Input the speech signal into the VGGish model and convert it into a 128-dimensional VGGish feature vector; Step 25: Classify and evaluate the importance of eGeMAPS features through machine learning techniques, and directly use deep learning networks to process and analyze VGGish features.

6. The method for auxiliary diagnosis of depression based on speech signals according to claim 2, characterized in that: In step 3, the eGeMAPS and VGGish features are processed and fused, and the encoder based on the Transformer architecture is used to build the EGV-AttNet model, which specifically includes the following steps: Step 31: Use feature importance curves to weight eGeMAPS features, implement dynamic weighting based on the relevance of the task objectives, and enhance the contribution and identifiability of features in subsequent model training; Step 32: Use a one-dimensional convolutional layer to preprocess the VGGish features to achieve spatial encoding of primary features, and input these encoded features into a series of residual network structures; Step 33: Input the residual block output into the ECA-Net attention mechanism model. By considering the cross-dependency between feature channels, a channel attention is implemented to dynamically adjust the response strength of the feature channel, highlight the most discriminative features, and suppress redundant or irrelevant information, thereby improving the model's ability to extract key features. Step 34: Input the ECA-Net output into the LSTM network, so that the model can focus on important events within the duration when processing the audio stream and exclude uninfluential signal changes; Step 35, perform feature conversion on the eGeMAPS and VGGish features processed in the above steps and input them into the feature fusion layer; Step 36: Input the output of the feature fusion layer into the encoder based on the Transformer architecture for further encoding, and perform classification prediction through the fully connected layer. The calculation expression of the multi-head attention mechanism in the Transformer architecture is: Among them, Q, K, V represent query, key and value respectively, and dk represents the dimension of the key.

7. The method for auxiliary diagnosis of depression based on speech signals according to claim 6, characterized in that: In step 32, the residual network structure is as follows: Among them: x is the residual network input, Y is the residual network output, represents the operation of the convolutional layer, and wi is the weight of the convolutional layer.

8. The method for auxiliary diagnosis of depression based on speech signals according to claim 6, characterized in that: In step 34, the LSTM network structure is as follows: i t =σ(w xi x t +w hi h t-1 +w ci c t-1 +b i ) f t =σ(w xf x t +w hf h t-1 +w cf c t-1 +b f ) c t =f t c t-1 +i t tan(w xc x t +w hc h t-1 +b c ) o t =σ(w xo x t +w ho h t-1 +w co c t-1 +b o ) the t =oh t tank t Where: x t Input for the current time, h t-1 is the external state at the previous moment, c t is the memory cell state, c t-1 is the memory cell state at the previous moment, b is the offset matrix, w x For x t The weight matrix, w h h t-1 The weight matrix, w c c t-1 The weight matrix, i t is the output of the input gate, f t is the output of the forget gate, o t is the output of the output gate, y t is the output value, and σ adopts the Sigmoid function.

9. The method for auxiliary diagnosis of depression based on speech signals according to claim 1, characterized in that: In step 4, the EGV-AttNet model is trained and saved, and the new speech signal is diagnosed. The output of the diagnosis result includes the following steps: Step 41, using a stratified sampling method to divide the inter-group control data set into a training set, a validation set, and a test set in a ratio of 8:1:1; Step 42: Select the weighted cross entropy loss function as the loss function for training; Step 43: Select Adam as the optimizer for model training. Step 44: Introduce a learning rate scheduler using the ReduceLROnPlateau strategy. When the validation set loss stops improving, the learning rate is automatically reduced to further optimize the training process. Step 45: Use the Optuna framework to automatically and efficiently search for the optimal hyperparameter combination; Step 46: During the hyperparameter optimization process, a pruning strategy and an early stopping mechanism are used to reduce unnecessary computing resource consumption and improve search efficiency. Step 47: Based on the prediction results on the validation set, select and save the best performing model, test it on the test set, output the diagnosis results, and evaluate the generalization ability of the model; Step 48: Evaluate the diagnosis results to ensure their accuracy and reliability; considering the performance difference between the current model on the validation set and the test set, use five-fold cross validation to carefully evaluate the performance of the model; Step 49: Use the confusion matrix as the analysis object to evaluate the performance of the model in identifying positive and negative samples on the validation set and the test set.

10. The method for auxiliary diagnosis of depression based on speech signals according to claim 9, characterized in that: In step 48, the diagnosis results include accuracy, precision, recall and F1 score indicators.

Images