A method for analyzing speech phenotype of depression based on SHAP value

Abstract

The present invention relates to the field of speech signal processing, and in particular to a method for analyzing speech phenotypes of depression based on SHAP values, wherein the audio in the collected data set is preprocessed; the preprocessed audio is subjected to frame windowing processing; a segmented feature statistical method is used to perform mid-term feature extraction on each audio file to obtain a feature vector matrix D; the obtained feature vector matrix D and the disease identification label of depression are used as input feature matrices to train an SVM classifier and generate an SVM model; the SHAP values ​​of each feature value are calculated to obtain speech feature markers in depression classification, and finally determine the speech phenotype of depression classification according to the speech phenotype corresponding to the speech feature marker. The method extracts a digital phenotype of sound features with clinical diagnosis, prediction, disease assessment, and efficacy evaluation significance, and constructs a new digital phenotype feature marker of depressive mental illness.

Description

Technical Field

[0001] The present invention relates to the field of speech signal processing, and in particular to a method for analyzing speech phenotypes of depression based on SHAP values. Background Art

[0002] Depression is a mental illness. Timely and effective intervention and treatment of depression can help reduce the occurrence of depression, reduce social burden, alleviate patient suffering, and reduce family burden.

[0003] Speech contains acoustic and linguistic features that can be recognized by machine learning models to measure the behavioral health of the speaker. In addition, speech data collection is relatively simple and attractive for patients, and relatively cheap and convenient, requiring only a microphone, a quiet place, and a device to collect audio samples. Therefore, speech-based biomarkers can screen for diseases in advance, monitor disease progression and response to treatment, and serve as useful surrogate markers for clinical research with informed consent.

[0004] Feature selection methods have been used in depression modeling studies with the goal of improving the accuracy of depression diagnosis. However, these studies did not report the selected feature sets, which would improve the understanding of the generalizability of their findings, nor did they report stability measures and procedures to increase it. In addition, some of these studies used feature transformation methods that fail to identify the actual features that contribute to modeling.

[0005] Accordingly, a new method for analyzing depression phenotype is needed in the art to solve the above problems. Summary of the invention

[0006] Based on the above situation, the present invention provides a method for analyzing the speech phenotype of depression based on SHAP value, which extracts the digital phenotype of sound characteristics with clinical diagnosis, prediction, disease assessment, and efficacy evaluation significance, and constructs a new characteristic marker of the digital phenotype of depressive disorder mental illness.

[0007] The present invention provides the following technical solution: a method for analyzing speech phenotype of depression based on SHAP value, comprising the following steps:

[0008] S1, preprocessing the audio in the collected data set;

[0009] S2, performing frame division and windowing processing on the pre-processed audio;

[0010] S3, using segmented feature statistics to extract mid-term features from each audio file, and obtain feature vector matrix D; this solves the problem of different durations of each audio file and the extraction of mid-term features;

[0011] S4, using the obtained feature vector matrix D and the disease identification label of depression as input feature matrix, training the SVM classifier, and generating an SVM model;

[0012] S5, using the prediction function f(x) of the trained SVM model and the extracted eigenvector matrix D as input parameters for SHAP value calculation, and calculating the SHAP value of each eigenvalue;

[0013] S6. Analyze the importance of features through SHAP values, take the statistical average of the top 20 features in each model, and finally sort each feature according to the average SHAP value. According to the sorting results, select the top-ranked features as the speech feature markers in the depression classification of this dataset, and finally determine the speech phenotype of depression classification according to the speech feature markers.

[0014] In step S1, during pre-processing, silent audio, audio shorter than 1 second, and silent segments at the beginning and end of each recording are deleted to eliminate background noise.

[0015] In step S2, the frame size of audio short-time feature extraction is 50ms. In order to reduce spectrum leakage, the original audio signal is multiplied by the window function, and the window function is set to the Hamming window. The calculation formula is as follows:

[0016]

[0017] Where W(n) is the window coefficient at sample index n and N is the total number of samples in the window.

[0018] In step S3, the medium-term window is set to 1s. For each audio file, 34 short-term feature sequences are extracted on each short-term frame and combined into a 34-dimensional short-term feature matrix called A. The 34-dimensional short-term feature matrix A corresponds to an incremental feature matrix ΔA.

[0019] ΔA=χ1-χ0

[0020] Where χ1 represents the eigenvalue of the current short-time frame corresponding to A, χ0 represents the eigenvalue of the previous short-time frame, and the incremental eigenvalue of the current short-time frame of ΔA can be obtained by subtracting the eigenvalue of the previous short-time frame from the current eigenvalue;

[0021] Combine A and ΔA into a matrix X with a dimension of 68. Extract the mean and standard deviation of X in each metaphase window to obtain new feature vector matrices B and C as metaphase segment feature statistics.

[0022] The following two formulas are the formulas for calculating the mean and standard deviation, respectively. In the formula, x represents the eigenvector of each short-term feature, n is the number of eigenvectors, that is, the number of short-term frames, and each eigenvalue is added and divided by the number of frames to get the mean of the feature. In the formula N is the number of short-term frames, and the standard deviation σ(r) is obtained;

[0023]

[0024]

[0025] The B and C feature vector matrices of each mid-term window are averaged and combined into a long-term feature vector matrix D of the audio file, which has a dimension of 136.

[0026] f(x) is the prediction function of SVM. Its input is a 138-dimensional audio vector of an audio, and its output is the probability value of depression and non-depression. The goal of SHAP is to explain the model judgment results by calculating the contribution of each feature in x to the prediction. This method combines the viewpoints of Shapley Value and LIME.

[0027] In step S5, when calculating the SHAP value, the following steps are included:

[0028] S51, the dimension of the feature matrix D is k, and k indicator vectors with values ​​of 0 or 1 are taken to form a vector z';

[0029] S52, define conversion function h x (), used for retaining the original eigenvalues ​​at the positions of 1 according to the indicator vector for the input audio eigenvector matrix D, and replacing the positions of 0 with the eigenvalues ​​of randomly selected samples;

[0030] S53, pass the indicator vector z' through h x () is transformed into a feature vector x';

[0031] S54, calculate the predicted value f(h) corresponding to the feature vector x' x (z′)), the generated sample prediction values ​​and samples constitute the data set for training the linear explanation model g();

[0032] S55. Calculate the corresponding weight for z', where M is the number of all features and |z'| is the number of 1s in z';

[0033] S56, the linear model g(x) is fitted by the following formula, and the linear model fitting L(f,g,π x′ ) is the SHAP value,

[0034]

[0035]

[0036] From the above description, it can be seen that this scheme constructs a new characteristic marker of the digital phenotype of depressive disorder mental illness through the obtained audio. Only a microphone, a quiet place and a device for collecting audio samples are needed to obtain the audio, which is helpful for pre-screening. The present invention proposes a method for extracting a digital phenotype of sound features with clinical diagnosis, prediction, disease assessment, and efficacy evaluation significance to construct a new characteristic marker of the digital phenotype of depressive disorder mental illness. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of a specific implementation mode of the present invention.

[0038] Figure 2 This is the result diagram of the SVM model after training.

[0039] Figure 3 This is the result graph of feature SHAP value ranking of the SVM model trained with the maximum training set.

[0040] Figure 4 This is a graph showing the relationship between features and SHAP values. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the specific embodiments of the present invention to clearly and completely describe the technical solutions in the specific embodiments of the present invention. Obviously, the specific embodiment described is only one specific embodiment of the present invention, not all specific embodiments. Based on the specific embodiments of the present invention, all other specific embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] As can be seen from the accompanying drawings, the present scheme 1, a method for analyzing the speech phenotype of depression based on SHAP values, is characterized by comprising the following steps:

[0043] S1, preprocessing the audio in the collected data set;

[0044] During preprocessing, remove silent audio, audio shorter than 1 second, and silent segments at the beginning and end of each recording to eliminate background noise.

[0045] S2, performing frame division and windowing processing on the pre-processed audio;

[0046] The frame size of audio short-time feature extraction is 50ms. In order to reduce spectrum leakage, the original audio signal is multiplied by the window function, and the window function is set to the Hamming window. The calculation formula is as follows:

[0047]

[0048] Where W(n) is the window coefficient at sample index n and N is the total number of samples in the window.

[0049] S3, using segmented feature statistics to extract mid-term features from each audio file to obtain a feature vector matrix D;

[0050] In step S3, the medium-term window is set to 1s. For each audio file, 34 short-term feature sequences are extracted on each short-term frame and combined into a 34-dimensional short-term feature matrix called A. The 34-dimensional short-term feature matrix A corresponds to an incremental feature matrix ΔA.

[0051] ΔA=χ1-χ0

[0052] Where χ1 represents the eigenvalue of the current short-time frame corresponding to A, χ0 represents the eigenvalue of the previous short-time frame, and the incremental eigenvalue of the current short-time frame of ΔA can be obtained by subtracting the eigenvalue of the previous short-time frame from the current eigenvalue;

[0053] Combine A and ΔA into a matrix X with a dimension of 68. Extract the mean and standard deviation of X in each metaphase window to obtain new feature vector matrices B and C as metaphase segment feature statistics.

[0054] The following two formulas are the formulas for calculating the mean and standard deviation, respectively. In the formula, x represents the eigenvector of each short-term feature, n is the number of eigenvectors, that is, the number of short-term frames, and each eigenvalue is added and divided by the number of frames to get the mean of the feature. In the formula N is the number of short-term frames, and the standard deviation σ(r) is obtained;

[0055]

[0056]

[0057] The B and C feature vector matrices of each mid-term window are averaged and combined into a long-term feature vector matrix D of the audio file, which has a dimension of 136.

[0058] S4, using the obtained feature vector matrix D and the disease identification label of depression as input feature matrix, training the SVM classifier, and generating an SVM model;

[0059] S5, using the prediction function f(x) of the trained SVM model and the extracted eigenvector matrix D as input parameters for SHAP value calculation, and calculating the SHAP value of each eigenvalue;

[0060] The calculation of SHAP value includes the following steps:

[0061] S51, the dimension of the feature matrix D is k, and k indicator vectors with values ​​of 0 or 1 are taken to form a vector z';

[0062] S52, define conversion function h x (), used for retaining the original eigenvalues ​​at the positions of 1 according to the indicator vector for the input audio eigenvector matrix D, and replacing the positions of 0 with the eigenvalues ​​of randomly selected samples;

[0063] S53, pass the indicator vector z' through h x () is transformed into a feature vector x';

[0064] S54, calculate the predicted value f(h) corresponding to the feature vector x' x (z′)), the generated sample prediction values ​​and samples constitute the data set for training the linear explanation model g();

[0065] S55. Calculate the corresponding weight for z', where M is the number of all features and |z'| is the number of 1s in z';

[0066] S56, the linear model g(x) is fitted by the following formula, and the linear model fitting L(f,g,π x′ ) is the SHAP value,

[0067]

[0068]

[0069] S6. Analyze the importance of features through SHAP values, take the statistical average of the top 20 features in each model, and finally sort each feature according to the average SHAP value. According to the sorting results, select the top-ranked features as the speech feature markers in the depression classification of this dataset, and finally determine the speech phenotype of depression classification according to the speech feature markers.

[0070] When conducting specific analysis, the dataset used was the EATD dataset, which is composed of audio and text excerpts from interviews with 162 volunteers. There are 30 depressed volunteers and 132 non-depressed volunteers in EATD-Corpus. The total duration of the response audio in the dataset is about 2.26 hours. Since the purpose of the experiment is not to pursue higher classification accuracy, but to analyze the importance of features in the depression classification process, a relatively balanced sample selection was adopted. Finally, all the depression audios were selected in the dataset, and the same number of non-depression audios were randomly selected as the experimental dataset.

[0071] The audio in the dataset is preprocessed. First, silent audio, audio shorter than 1 second, and silent segments at the beginning and end of each recording are deleted to eliminate background noise.

[0072] The frame size of audio short-term feature extraction is 50ms. In order to solve the problem of different durations of each audio file, we use segmented feature statistics. According to our assumption, the severity of depression remains unchanged over a period of time, rather than changing at every moment. Therefore, we set the medium-term window length to 1s, obtain the mean and standard deviation of short-term audio features, obtain a feature statistics vector for each medium-term window, and finally perform long-term averaging on the entire audio segment.

[0073] 34 commonly used short-term features and corresponding incremental features are extracted on the short-term frame. In this case, the total number of features extracted for each short-term frame is 68. Then the mean and standard deviation of each short-term feature sequence are extracted in the medium-term window. Finally, the feature vector we extracted has a total of 136 values. The 34 short-term features are shown in Table 1.

[0074] Table 1 34 short-term characteristics

[0075]

[0076] The feature matrix extracted from the audio and the corresponding label combination are used as input to train the SVM classifier, and the Gaussian kernel is selected as the kernel function of the SVM.

[0077] In order to eliminate the problem of classification model sensitivity in small sample data training, so that the machine learning model can better learn the rules of depression classification in limited data, so that the calculated SHAP value is more closely related to depression classification, the training set of the machine learning model is divided into seven training subset classes according to the scale for training, and the corresponding model results are obtained, and feature importance analysis is performed on these models.

[0078] In the model evaluation part, common evaluation indicators for classification are selected, including F1 score, recall rate, and precision rate.

[0079] The training data is divided into seven training subsets of different sizes to train the SVM. 168 speech features are extracted from each audio sample.

[0080] The results of the SVM model after training with the largest number of training subsets as input are as follows Figure 2 As shown in the figure, the classification accuracy of the training data obtained by training with the largest training subset is 79.3%, and the average classification accuracy of the model obtained by training with the other six training subsets is 78.5%.

[0081] Kernel SHAP is a method that uses a special weighted linear regression to calculate the importance of each feature. The importance values ​​calculated are the Shapley values ​​from game theory and the coefficients from local linear regression. The basic idea is to calculate the marginal contribution of a feature when it is added to the model, and then take into account the different marginal contributions of the feature in all feature sequences and take the average, which is the SHAP value of the feature.

[0082] The prediction function P(x) of SVM takes as input a 138-dimensional audio vector of an audio track and outputs the probability value of depression or non-depression.

[0083] The P(x) function of the trained SVM model and the extracted eigenvector matrix are used as input parameters of the SHAP interpreter to calculate the SHAP value of the eigenvalue, and then the importance of the input feature is calculated by summing up the average value.

[0084] For the SVM models trained under different data scales, SHAP value analysis is used to analyze the feature importance. Take the feature SHAP value ranking results of the SVM model trained with the largest training set as an example. Figure 3 As shown,

[0085] In the context of the dataset used in this example, we calculated the SHAP values ​​for seven SVM models and averaged them. We found that the importance of the top 11 features was significantly higher than that of the remaining features. Finally, we obtained 11 features that were closely related to depression classification as characteristic markers of depressive disorders in the above dataset, as shown in Table 2. Among them, the 12th dimensional differential chromaticity standard deviation feature showed far-leading importance in different models.

[0086] Table 2 Eleven characteristics closely related to depression classification

[0087]

[0088] For these 11 features, we perform feature model dependency analysis. The x-axis is the feature value, and the y-axis is the SHAP value of the corresponding feature. We take the feature with a feature sequence of 134 as an example and draw the results as follows: Figure 4 shown.

[0089] Through feature model dependency analysis, we found that only the average value of the five-dimensional chromaticity coefficient and the average value of the five-dimensional MFCC coefficient were positively correlated with depression tendency, while the rest were negatively correlated with depression tendency. Through observation, it was found that the above 11 features all belong to the coefficients of chromaticity coefficient and MFCC in different dimensions, obtained by difference or average and standard deviation.

[0090] The sounds that humans make are filtered through the shape of the vocal tract (tongue, teeth, etc.). This shape determines the sounds that are made. If we can accurately determine the shape, this should accurately represent the phonemes that are produced. The shape of the vocal tract is reflected in the envelope of the short-time power spectrum, and the job of MFCCs is to accurately represent that envelope.

[0091] First, we analyze the MFCC calculation process. The speech is transformed through Fourier transform on a short time frame, passed through a Mel filter, then logarithm operation, and finally discrete cosine transform. Through these operations, we can separate the envelope from other details in the spectrum. We take the first thirteen dimensions as input features, where the one-dimensional coefficient reflects the energy size of the frame, and the other features are combined to describe the overall profile of the spectrum. The standard deviation of the MFCC coefficients in Table 2 has the meaning of the fluctuation size of speech energy, while the five-dimensional, seven-dimensional, and thirteen-dimensional MFCCs reflect the resonance peaks and the shape of the spectrum envelope, which also have an important impact on depression classification.

[0092] The chromaticity vector is usually a 12-element feature vector, indicating the energy of each pitch {C, C#, D, D#, E, F, F#, G, G#, A, A#, B} in the signal. Its calculation process is similar to MFCC. The frequency components of the speech are obtained by short-time Fourier transform, frequency filtering and peak detection are performed, and then the reference frequency is calculated relative to 440hz. The pitch mapping is performed using a weighting scheme with a cosine function. The maximum value is divided by the frame-by-frame normalization feature to eliminate the dependence on global loudness. Then we can get a result HPCP sequence. The differential standard deviation and standard deviation of the eight-dimensional and twelve-dimensional chromaticity coefficients in Table 2 and the differential standard deviation of the one-dimensional and nine-dimensional chromaticity coefficients can all represent the volatility of speech in a specific frequency range. The average value of the five-dimensional chromaticity coefficient in Table 2 represents the energy corresponding to the frequency of the pitch set corresponding to the chromaticity E. This can reflect the close connection between the frequency distribution of the pitch change and energy of speech over a certain period of time and the depression classification.

[0093] In summary, we conclude that the 11 features extracted above reflect that the speech's formant, spectrum envelope, energy fluctuation, pitch change, and energy distribution in frequency are closely related to the classification model of depression. Combined with the feature extraction process, we speculate that this shows that the tone change of speech and the richness of energy change play an important role in depression classification. And this phenotype is used as the depression classification speech phenotype under this data.

[0094] In this specific implementation, the data is decomposed into a training set and a test set in advance. The speech data in the training set is processed to obtain 165 speech acoustic features. The SVM classifier is trained with different amounts of data to reduce the individual differences of small samples. Then, the SHAP value of the features is calculated for different model results using weighted linear regression. Finally, 11 features closely related to depression classification are obtained through statistical analysis. We use these features as speech biomarkers for depression classification on this data set, and then perform model dependency analysis on these features to summarize the relevance of these features to depression classification. Finally, through the feature extraction process and practical significance, it is found that the speech spectrum envelope and the energy changes of speech, the changes in pitch are closely related to depression classification. The analysis shows that the richness of the changes in the volume and tone of the patient during speech expression can be used as the speech phenotype of depression classification on this data.

[0095] It can be seen from the accompanying drawings that although the specific embodiments of the present invention have been shown and described, it is understandable to those skilled in the art that various changes, modifications, substitutions and variations may be made to these specific embodiments without departing from the principles and spirit of the invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for analyzing speech phenotypes of depression based on SHAP values, characterized in that The steps include: S1, preprocessing the audio in the collected data set; S2, performing frame division and windowing processing on the pre-processed audio; S3, using segmented feature statistics to extract mid-term features from each audio file to obtain a feature vector matrix D; In step S3, the medium-term window is set to 1s. For each audio file, 34 short-term feature sequences are extracted on each short-term frame and combined into a 34-dimensional short-term feature matrix called A. The 34-dimensional short-term feature matrix A corresponds to an incremental feature matrix ΔA. in represents the eigenvalue of the current short-time frame corresponding to A, Represents the eigenvalue of the previous short-time frame. Subtracting the eigenvalue of the previous short-time frame from the current eigenvalue can obtain the incremental eigenvalue of the current short-time frame ΔA; Combine A and ΔA into a matrix X with a dimension of 68. Extract the mean and standard deviation of X in each metaphase window to obtain new feature vector matrices B and C as metaphase segment feature statistics. The following two formulas are the formulas for calculating the mean and standard deviation, respectively. In the formula, x represents the eigenvector of each short-term feature, n is the number of eigenvectors, that is, the number of short-term frames, and each eigenvalue is added and divided by the number of frames to get the mean of the feature. , where r = , N is the number of short-term frames, and the standard deviation is obtained ; The B and C feature vector matrices of each mid-term window are averaged and combined into the long-term feature vector matrix D of the audio file, and the dimension of the matrix is ​​136; S4, using the obtained feature vector matrix D and the disease identification label of depression as input feature matrix, training the SVM classifier, and generating an SVM model; S5, using the prediction function f(x) of the trained SVM model and the extracted eigenvector matrix D as input parameters for SHAP value calculation, and calculating the SHAP value of each eigenvalue; S6. Analyze the importance of features through SHAP values, take the statistical average of the top 20 features in each model, and finally sort each feature according to the average SHAP value. According to the sorting results, select the top-ranked features as the speech feature markers in the depression classification of this dataset, and finally determine the speech phenotype of depression classification according to the speech feature markers.

2. The method for analyzing depression speech phenotype based on SHAP value according to claim 1, characterized in that: In step S1, during pre-processing, silent audio, audio shorter than 1 second, and silent segments at the beginning and end of each recording are deleted to eliminate background noise.

3. The method for analyzing depression speech phenotype based on SHAP value according to claim 1 or 2, characterized in that: In step S2, the frame size of audio short-time feature extraction is 50ms. In order to reduce spectrum leakage, the original audio signal is multiplied by the window function, and the window function is set to the Hamming window. The calculation formula is as follows: ,0≤n≤N Where W(n) is the window coefficient at sample index n and N is the total number of samples in the window.

4. The method for analyzing depression speech phenotype based on SHAP value according to claim 1, characterized in that: The calculation of the SHAP value involves the following steps: S51, the dimension of the feature matrix D is k, and k indicator vectors with values ​​of 0 or 1 are taken to form a vector z'; S52, define conversion function , used for retaining the original eigenvalues ​​at the positions of 1 and replacing the positions of 0 with the eigenvalues ​​of randomly selected samples according to the indicator vector for the input audio eigenvector matrix D; S53, the indication vector z' is passed Transformed into feature vector x'; S54, calculate the predicted value corresponding to the feature vector x' The generated sample prediction values ​​and samples constitute the data set for training the linear explanation model g(); S55. Calculate the corresponding weight for z', where M is the number of all features and |z'| is the number of 1s in z'; S56, the linear model g(x) is fitted by the following formula, is the SHAP value, 。

Images