Method and System for Detecting Speech Depression Level Based on Complex Spectrum

Abstract

The present invention discloses a method for detecting speech depression level based on complex spectrum, comprising the following steps: S1: Extracting the complex spectrum of the original speech waveform as the input feature for predicting the speech depression level; S2: Using a deep speech feature extractor to model the mutual dependence between channels of the features to obtain discriminative acoustic features; S3: Using a two-layer convolutional neural network to compress the frequency channels into one dimension to obtain deep speech features with only the time dimension; S4: Using attention time pooling to obtain the long-term correlation of low-dimensional speech representations, and dynamically selecting more important information according to the attention mechanism to obtain corresponding prediction results. A system for detecting speech depression level based on complex spectrum is also disclosed. The present invention first uses complex spectrum for detecting speech depression level, which can significantly improve the accuracy of detecting speech depression level.

Description

Technical Field

[0001] The present invention relates to the technical field of speech depression detection, and particularly to a method and system for detecting speech depression level based on complex spectrum. Background Art

[0002] Depression is a common mental disorder, and speech signals can reflect people's emotions and stress. In addition, physiological studies have also shown that there are differences in speech signals between depressed individuals and normal individuals. Based on the above research results, many works on predicting depression levels based on speech signals have been proposed. Automatic speech depression level prediction (SDLP) is a very challenging problem in affective computing. Many studies have achieved quite good performance for SDLP. However, most of the input speech features in these studies are based on the magnitude spectrum, thus losing the phase spectrum information. Therefore, these speech features may lose some important information related to depression.

[0003] Representative deep speech features can improve the prediction performance. However, many features are based on the magnitude spectrum, which loses the phase spectrum information. The phase spectrum may contain a lot of important information for SDLP. Many studies have shown that the phase spectrum is very important for speech quality and intelligibility. Therefore, it is unreasonable to discard this information. The current work mainly focuses on two aspects: 1) improving the acoustic features of audio; 2) designing new classification models. The magnitude spectrum and the phase spectrum are two basic acoustic features obtained based on the Fourier transform, reflecting different characteristics of audio. In early studies, mainly the magnitude spectrum was processed while ignoring the phase spectrum information. The phase information is very important for the speech depression detection task. Compared with the magnitude spectrum, the phase spectrum may contain more important information for detecting the speech depression level. Therefore, how to effectively utilize the speech feature information is a challenging problem.

[0004] Therefore, there is an urgent need to provide a new method for detecting speech depression level to solve the above problems. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for detecting speech depression level based on complex spectrum, which can significantly improve the accuracy of speech depression detection technology.

[0006] To solve the above technical problem, a technical solution adopted by the present invention is: to provide a method for detecting speech depression level based on complex spectrum, including the following steps:

[0007] S1: Extract the complex spectrum of the original speech waveform as the input feature for automatic speech depression level prediction;

[0008] S2: Model the mutual dependence between channels of features using a deep voice feature extractor to obtain discriminative acoustic features;

[0009] S3: Compress the frequency channels into one dimension using a two - layer convolutional neural network to obtain deep voice features with only the time dimension;

[0010] S4: Use attention time pooling to obtain the long - term correlation of low - dimensional voice representations, and dynamically select more important information according to the attention mechanism to obtain the corresponding prediction results.

[0011] In a preferred embodiment of the present invention, the specific steps of step S1 include:

[0012] S101: Convert the time - domain voice signal into the time - frequency domain using the short - time Fourier transform STFT:

[0013] Real[t, f]+i*Imag[t, f]=STFT(x[k]) (1)

[0014] where x[k] represents the voice signal in the time domain, k is the time index of the voice signal, Real and Imag are the corresponding real and imaginary parts of STFT, t is the index of the number of time frames, and f is the index of the frequency unit;

[0015] S102: Concatenate the real part and the imaginary part of the STFT together to obtain the required complex - spectrum features, expressed as:

[0016]

[0017] where stack represents the concatenation operation, F and T are the frequency and the number of time frames respectively.

[0018] In a preferred embodiment of the present invention, the specific steps of step S2 include:

[0019] S201: First, compress the input vector using global pooling to obtain global information embedding:

[0020] z c =F GP (u c ) (3)

[0021] where u c is the input vector of the c - th channel U = [u1, u2,......, u c , F GP represents the GP operation, Z c is the c - th element of Z, and Z represents the global information;

[0022] S202: Apply two fully connected layers, make full use of the aggregated information through the squeezing operation, and fully capture the channel dependencies:

[0023] S = σ(W2δ(W1Z)) (4)

[0024] where W1 and W2 are the weight matrices of these two FC layers, δ and σ represent the ReLU and sigmoid functions respectively, and S represents the channel weights assigned according to importance;

[0025] S203: The scaling operation obtains the final output of the SE calculation block:

[0026]

[0027] where F scale represents channel multiplication.

[0028] In a preferred embodiment of the present invention, the specific steps of step S3 include:

[0029]

[0030] where V = [V1, V2,......, V T′ represents the deep voice features, T′ represents the time dimension after the deep voice feature extractor, and D represents the dimension of V i .

[0031] In a preferred embodiment of the present invention, the specific steps of step S4 include:

[0032]

[0033]

[0034] V′ = W[concate{μ(α i V i );ζ(α i V i )}] (9)

[0035] where V i represents a set of local descriptors of the deep voice features, V′ represents a single global descriptor, μ represents the mean, ζ represents the variance, and W represents the linear projection.

[0036] To solve the above technical problems, another technical solution adopted by the present invention is: to provide a speech depression level detection system based on complex spectra, including:

[0037] A speech feature input module, which is used to extract the complex spectrum of the original speech waveform as the input feature of the SDLP;

[0038] The complex spectral feature processing module is used to model the mutual dependence between channels of features by using a deep speech feature extractor to obtain discriminative acoustic features; compress the frequency channels into one dimension by using a two-layer convolutional neural network to obtain deep speech features with only the time dimension.

[0039] The attention time pooling module is used to obtain the long-term correlation of low-dimensional speech representations by using attention time pooling, and dynamically select more important information according to the attention mechanism to obtain corresponding prediction results.

[0040] The beneficial effects of the present invention are as follows:

[0041] (1) The present invention proposes a complex spectral squeezing and excitation network (CSENet) for SDLP. This network uses complex spectra as input features, and the complex spectra contain all speech information, including the amplitude spectrum and the phase spectrum. Therefore, applying this feature can make full use of speech information to improve the prediction accuracy.

[0042] (2) In addition, in order to obtain discriminative features for SDLP, a deep speech feature extractor (SE-ResNet) is used to model the mutual dependence between channels of features to obtain discriminative acoustic features, thereby improving the quality of deep speech representations.

[0043] (3) Finally, attention time pooling is used to obtain the long-term correlation of low-dimensional speech representations, and more important information is dynamically selected according to the attention mechanism. Attention time pooling is very effective for distinguishing different depression levels.

[0044] (4) The present invention is very helpful for the detection of speech depression levels and can significantly improve the accuracy of speech depression detection technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flowchart of the method for detecting speech depression level based on complex spectra according to the present invention;

[0046] Figure 2 is a schematic diagram of the CSENet model of the method and system for detecting speech depression level based on complex spectra;

[0047] Figure 3 is a block diagram of the structure of the system for detecting speech depression level based on complex spectra. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] The following elaborates on the preferred embodiments of the present invention in conjunction with the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0049] Please refer to Figure 1 andFigure 2 , embodiments of the present invention include:

[0050] A method for detecting the depression level of speech based on complex spectrum, comprising the following steps:

[0051] S1: Extract the complex spectrum (including amplitude spectrum and phase spectrum) of the original speech waveform as the input feature for automatic speech depression level prediction (SDLP);

[0052] The steps for extracting the complex spectrum features of the original speech waveform include the following:

[0053] S101: Use the short-time Fourier transform STFT to convert the time-domain speech signal into the time-frequency domain:

[0054] Real[t, f]+i*Imag[t, f] = STFT(x[k]) (1)

[0055] where x[k] represents the speech signal in the time domain, k is the time index of the speech signal, Real and Imag are the corresponding real and imaginary parts of the STFT, t is the index of the time frame number, and f is the index of the frequency unit;

[0056] Preferably, each audio sample is divided into several segments, each segment is 3s long, and the overlap rate between adjacent segments is 50%. The sampling rate of all generated speech waveforms is 8000Hz. For STFT, the size of the Hamming window is 50ms, the window offset is 12.5ms, and the number of points for the fast Fourier transform (FFT) is 512. Therefore, the size of the spectrogram is 257.

[0057] S102: Concatenate the real part and the imaginary part of the STFT together to obtain the required complex spectrum features, expressed as:

[0058]

[0059] where stack represents the concatenation operation, and F and T are the frequency and the number of time frames respectively.

[0060] S2: Use a deep speech feature extractor to model the mutual dependence between the channels of the features to obtain discriminative acoustic features; as Figure 2 shown in Deep Speech Feature Extraction, the SE calculation block includes a global pooling layer, two fully connected layers (FC), and a scaling operation (Scale).

[0061] S201: First, use global pooling (GP) to compress the input vector to obtain the embedding of global information (all information of the features):

[0062] z c = F GP (u c ) (3)

[0063] where u c is the c-th channel input vector U = [u1, u2,......, u c (generated by the convolution operation of the complex spectrum). F GP represents the GP operation, Z c is the c-th element of Z, and Z represents the global information.

[0064] S202: Apply two fully connected (FC) layers to make full use of the aggregated information (i.e., the global information Z obtained in step S201) through the squeezing operation (as shown in formula 4) and fully capture the channel dependencies:

[0065] S = σ(W2δ(W1Z)) (4)

[0066] where W1 and W2 are the weight matrices of these two FC layers. In addition, δ and σ represent the ReLU and sigmoid functions respectively, and S represents the channel weights assigned according to the importance.

[0067] S203: The scaling operation (Scale) obtains the final output of the SE calculation block:

[0068]

[0069] where F scale represents channel multiplication.

[0070] It should be noted that the deep voice feature extractor compresses the input vector through global pooling (GP) to obtain the global information (all the information of the features), and then uses the global information, applies two fully connected layers (FC) to obtain the importance of each feature channel, assigns channel weights according to the importance, improves the useful features, and suppresses the features that are not useful for the current task, thereby improving the quality of the deep voice representation.

[0071] S3: Compress the frequency channels into one dimension using a two-layer convolutional neural network to obtain the deep voice features with only the time dimension;

[0072]

[0073] where V = [V1, V2,......, V T′ represents the deep voice features, T' represents the time dimension after the deep voice feature extractor, and D represents the dimension of V i .

[0074] S4: Use the attention time pooling to obtain the long-term correlation of the low-dimensional speech representation, and dynamically select more important information (discriminative information related to the depression level) according to the attention mechanism to obtain the corresponding prediction result.

[0075]

[0076]

[0077] V′ = W[concate{μ(α i V i );ζ(α i V i )}] (9)

[0078] where V i represents a set of local descriptors of the deep speech features, V′ represents a single global descriptor, μ represents the mean, ζ represents the variance, and W represents the linear projection.

[0079] It should be noted that in step S2, the structure of the deep speech feature extractor is based on ResNet18, and 256 is used as the dimension of the deep speech features, which means D is 256. In addition, the extractor uses Adam as the optimizer, with a learning rate of 0.002 for AVEC2013 and a learning rate of 0.0006 for AVEC 2014. The complex spectral features are used as the input of the deep neural network classifier, and the deep neural network classifier is set for a certain number of training epochs for training. Finally, the best model in the training is selected for testing, and the obtained test result is used as the corresponding prediction result.

[0080] In the embodiment of the present invention, referring to Figure 3 , a voice depression level detection system based on complex spectra is further provided, including:

[0081] A voice feature input module, which is used to extract the complex spectrum (including the amplitude spectrum and the phase spectrum) of the original voice waveform as the input feature of the SDLP;

[0082] A complex spectrum feature processing module, which is used to adopt a deep speech feature extractor to model the mutual dependence between the channels of the features to obtain discriminative acoustic features; use a two-layer convolutional neural network to compress the frequency channels into one dimension to obtain deep speech features with only the time dimension;

[0083] An attention time pooling module, which is used to use the attention time pooling to obtain the long-term correlation of the low-dimensional speech representation, and dynamically select more important information (discriminative information related to the depression level) according to the attention mechanism to obtain the corresponding prediction result.

[0084] In the present invention, the Beck Depression Inventory-II (BDI-II) was used as the ground truth to represent the depression level, and experiments were conducted on two public datasets, AVEC 2013 and AVEC 2014. To quantitatively evaluate the SDLP results, the root mean square error (RMSE) and the mean absolute error (MAE) were used as evaluation metrics.

[0085] Table 1

[0086]

[0087] Table 2

[0088]

[0089] In the present invention, CSENet mainly consists of a complex spectrum, a deep speech feature extractor (SE-ResNet), and an attention time pooling.

[0090] As can be seen from Table 1, when using SE ResNet to replace ResNet as the deep feature extractor, the performance can be improved. Compared with the CResNet method, the proposed CSENet method can reduce the RMSE and MAE from 9.42 and 7.19 to 9.28 and 6.79 respectively. As can be seen from Table 2, although the RMSE performance of CSENet is worse than that of CResNet, the MAE result is still better than that of CResNet. These results indicate that the deep speech features extracted by SE-ResNet are more suitable for SDLP than ResNet. This is because SE-ResNet with squeeze and excitation operations can automatically obtain the importance of each feature channel. In addition, it can also improve useful features according to this importance and suppress features that are not useful for the current task. Therefore, CSENet can obtain better performance than CResNet.

[0091] Tables 1 and 2 also show the results of other SDLP methods. Among them, the [1,2,3,5,7,8,9] method uses LLDs and MFCCs as input speech features. [4] only uses MFCCs as input speech features. The magnitude spectrum is used as the input speech feature for [6]. These speech features lose the phase spectrum information, which may lose some very important information for SDLP. In the present invention, the complex spectrum is used as the speech feature, which includes the magnitude spectrum and the phase spectrum. It can be found from Table 1 that compared with other SDLP methods, the SDLP method based on the complex spectrum proposed in the present invention can obtain the best performance in terms of both RMSE and MAE. In addition, for the AVEC 2014 dataset, although the performance of the method proposed in the present invention is worse than that of [6] in terms of RMSE, it obtains better results in terms of the MAE evaluation index. For the AVEC 2013 dataset, the CSENet method proposed in the present invention obtains the state-of-the-art performance in terms of the MAE evaluation index.

[0092] These results prove the effectiveness of the method proposed in the present invention. In addition, these results also show that the complex spectrum is a suitable feature for the SDLP task. This is because the method proposed in the present invention is based on the complex spectrum and can make full use of the speech information.

[0093] The above are only the embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for detecting the depression level of speech based on complex spectrum, characterized in that, It includes the following steps: S1: Extract the complex spectrum of the original speech waveform, splice the real part and the imaginary part of the complex spectrum together to obtain complex spectrum features, which are used as input features for automatic speech depression level prediction; S2: Use a deep speech feature extractor to model the mutual dependence between channels of the features to obtain discriminative acoustic features; The specific steps include: S201: First, use global pooling to compress the input vector to obtain a global information embedding: z c = FG P (u c ) (3) Among them, u c is the input vector of the c-th channel, U = [u1, u2, ……, u c , F GP represents the GP operation, z c is the c-th element of Z, and Z represents the global information; S202: Apply two fully connected layers to make full use of the aggregated information through the squeezing operation and fully capture the channel dependence: S = σ(W2δ(W1Z)) (4) where W1 and W2 are the weight matrices of these two fully connected layers, δ and σ represent the ReLU and sigmoid functions respectively, and S represents the channel weights assigned according to the importance; S203: The scaling operation obtains the final output of the SE calculation block: Among them, F scale represents channel multiplication; S3: Use a two-layer convolutional neural network to compress the frequency channels into one dimension to obtain deep speech features with only the time dimension; the specific steps include: Among them, V = [V1, V2, ……, V T represents the deep voice feature, T′ represents the time dimension after the deep voice feature extractor, and D represents the dimension of V i . S4: Use attention time pooling to obtain the long-term correlation of low-dimensional speech representations, and dynamically select discriminative information related to the depression level according to the attention mechanism to obtain the corresponding prediction results.

2. The method for detecting speech depression level based on complex spectrum according to claim 1, wherein, The specific steps of step S1 include: S101: Use the short-time Fourier transform STFT to convert the time-domain speech signal into the time-frequency domain: Real[t, f] + i*Imag[t, f] = STFT(x[k]) (1) where x[k] represents the speech signal in the time domain, k is the time index of the speech signal, Real and Imag are the corresponding real and imaginary parts of the STFT, t is the index of the number of time frames, and f is the index of the frequency unit; S102: Splice the real part and the imaginary part of the STFT together to obtain complex spectrum features, which are expressed as: where stack represents the splicing operation, and F and T are the frequency and the number of time frames respectively.

3. A method for detecting the speech depression level based on the complex spectrum according to claim 1, characterized in that, The specific steps of step S4 include: d i = V i T V i (7) V′ = W[concatenate{μ(α i V i );ζ(α i V i ))] (9) Among them, V i represents a set of local descriptors of deep voice features, V' represents a single global descriptor, μ represents the mean, ζ represents the variance, and W represents a linear projection.

4. A voice depression level detection system based on complex spectrum, which adopts the voice depression level detection method based on complex spectrum as described in any one of claims 1 to 3, and is characterized in that, It includes: A speech feature input module, which is used to extract the complex spectrum of the original speech waveform as the input feature of the SDLP; A complex spectrum feature processing module, which is used to use a deep speech feature extractor to model the mutual dependence between channels of the features to obtain discriminative acoustic features; use a two-layer convolutional neural network to compress the frequency channels into one dimension to obtain deep speech features with only the time dimension; An attention time pooling module, which is used to use attention time pooling to obtain the long-term correlation of low-dimensional speech representations, and dynamically select discriminative information related to the depression level according to the attention mechanism to obtain the corresponding prediction results.

Images