Speech emotion recognition method and system based on attention time-frequency network

Abstract

The application discloses a speech emotion recognition method and system based on an attention time-frequency network, wherein the log-Mel spectrum features obtained through a sample feature extraction module are more robust, then the time-frequency attention network and the encoder are used to obtain features that are irrelevant to abnormal noise (such as background noise, speaker identity, language, etc.) but are related to emotion, and finally, the emotion classifier module is used to further enhance the emotion discriminability of the obtained features and the noise-irrelevant generalization, so that the application has better recognition effect and higher recognition rate.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a speech emotion recognition method and system based on attention time-frequency networks. Background Technology

[0002] The purpose of speech emotion recognition is to enable machines to determine the emotional state (such as fear, happiness, sadness, etc.) of humans through speech signals, and it has become a research hotspot in the fields of affective computing and human-computer interaction. Many research institutions at home and abroad (such as the MIT Media Lab in the United States, the GLAM Lab at Imperial College London in the United Kingdom, and the Institute of Automation of the Chinese Academy of Sciences) are dedicated to speech emotion recognition research and have begun to try to apply it to fields such as education and healthcare.

[0003] The current challenge in speech emotion recognition technology lies in the fact that speech data is easily affected by factors such as environmental noise, speaker, language, and recording equipment, resulting in significant differences in the distribution of emotional features and making the representation of emotional information exceptionally difficult. This severely restricts the generalization performance of speech emotion recognition methods and poses a significant challenge to robust speech emotion recognition. The time-frequency characteristics of speech signals are of great significance for the representation of speech emotions; therefore, how to mine the frequency and time domain information of speech for the extraction of emotional features is a key technical challenge that robust speech emotion recognition urgently needs to overcome. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a speech emotion recognition method and system based on attention time-frequency network, which has better recognition effect and higher recognition rate.

[0005] To address the aforementioned technical problems, this invention provides a speech emotion recognition method based on attention time-frequency networks, comprising the following steps:

[0006] (1) Obtain several emotional speech samples, perform short-time discrete Fourier transform to obtain spectral features, and use them as input to the network;

[0007] (2) Input the log-Mel spectrum features of the training samples into the attention-based frequency domain encoder to obtain weighted frequency domain speech emotion features;

[0008] (3) Input the weighted frequency domain speech emotion features into the attention-based time domain encoder to obtain the weighted time domain speech emotion features, i.e., the time frequency speech emotion features;

[0009] (4) The time-frequency speech emotion features are classified using an emotion classifier;

[0010] (5) Jointly train the attention-based frequency domain encoder, the attention-based time domain encoder, and the sentiment classifier to obtain the optimal model parameters;

[0011] (6) Input the emotional speech sample to be identified into the trained attention-based frequency domain encoder, attention-based time domain encoder and emotion classifier to obtain the identified speech emotion.

[0012] Preferably, in step (1), acquiring several emotional speech samples and performing short-time discrete Fourier transform to obtain spectral features, which are then used as input to the network, specifically includes the following steps:

[0013] (1-1) Perform pre-emphasis, framing, and windowing preprocessing on the denoised speech signal;

[0014] (1-2) The spectrum is obtained by performing a short-time discrete Fourier transform on the preprocessed speech signal;

[0015] (1-3) The spectrum is scaled by Mel scale through multiple Mel filter banks and then logarithmized to obtain the log-Mel spectrum features, which are used as the input of the network.

[0016] Preferably, in step (2), inputting the log-Mel spectral features of the training samples into the attention-based frequency domain encoder to obtain weighted frequency domain speech emotion features specifically includes the following steps:

[0017] (2-1) Input the log-Mel spectrum features of the training samples into the frequency domain attention network to obtain the frequency domain attention a. f ;

[0018] (2-2) Frequency domain attention a f After multiplying the frequency domain feature x with the original frequency domain feature x by the broadcast product, the average value of the result is obtained as the weighted frequency domain feature x′.

[0019] (2-3) The frequency domain attention network contains a convolution module, a product module and a summation module. Each convolution module contains a convolution module operation, each product module contains a broadcast product operation, and each summation module contains an addition operation.

[0020] Preferably, in step (3), the weighted frequency domain speech emotion features are input into the attention-based time domain encoder to obtain weighted time domain speech emotion features, i.e., time frequency speech emotion features, specifically including the following steps:

[0021] (3-1) Input the log-Mel spectrum features of the training samples into the temporal attention network to obtain the temporal attention a. t ;

[0022] (3-2) Apply temporal attention a t After multiplying the weighted frequency domain feature x′ by the broadcast product, and then weighting the weighted frequency domain feature x′ by channel, the average value is used to obtain the weighted time domain feature, i.e., the weighted time-frequency feature x″.

[0023] (3-3) The temporal attention network contains a convolution module, a product module and a summation module. Each convolution module contains a convolution module operation, each product module contains a broadcast product operation, and each summation module contains an addition operation.

[0024] Preferably, in step (4), the classification of time-frequency features by the emotion discriminator specifically includes the following steps:

[0025] (4-1) The weighted time-frequency feature x″ is used for sentiment classification by a sentiment discriminator;

[0026] (4-2) The sentiment discriminator is a two-layer fully connected network with hidden node dimensions of 512 and sentiment category number c.

[0027] Accordingly, a speech emotion recognition system based on attention time-frequency networks includes:

[0028] The feature extraction module is used to extract the log-Mel spectrum features of speech samples;

[0029] The frequency domain attention module is used to compute the frequency domain attention of Mel features;

[0030] The frequency domain encoder module is used to perform frequency domain encoding on the Mel features after frequency domain attention enhancement;

[0031] The temporal attention module is used to compute temporal attention for Mel features;

[0032] The time-domain encoder module is used to perform time-domain encoding on the frequency-domain features after time-domain attention enhancement;

[0033] The sentiment classifier module is used to classify sentiment categories based on time-frequency features.

[0034] The attention time-frequency network training module is used to establish a time-frequency joint learning network composed of a frequency domain attention network, a frequency domain encoder network, a time domain attention network, a time domain encoder network, and a sentiment classification network. The corresponding features of each path are used as input to train the attention time-frequency network.

[0035] The speech emotion recognition module is used to input the speech sample to be recognized into a trained attention time-frequency network to obtain the recognized speech emotion.

[0036] Preferably, the feature extraction module specifically includes: a framing and windowing unit, used to perform framing and windowing operations on the clean speech signal; a spectrum extraction unit, used to perform short-time discrete Fourier transform on the framed and windowed speech signal to extract the spectrum; and a log-Mel spectrum extraction unit, used to perform Mel-scale filtering and logarithmic operation on the extracted spectrum to obtain log-Mel spectrum features.

[0037] Preferably, the frequency domain attention module specifically includes: a frequency domain attention calculation unit, used to obtain the frequency domain attention of the Mel features of the speech sample; and a frequency domain attention broadcast product unit, used to obtain the frequency domain attention features of the speech sample. The frequency domain encoder module specifically includes: a frequency domain feature channel weighted averaging unit, used to weight the frequency domain attention features of the speech sample; and a frequency domain encoding unit, used to encode the weighted frequency domain attention features to obtain enhanced frequency domain features.

[0038] Preferably, the temporal attention module specifically includes: a temporal attention calculation unit, used to obtain the temporal attention of the Mel features of the speech sample; and a temporal attention broadcast product unit, used to obtain the temporal attention features of the speech sample. The temporal encoder module specifically includes: a temporal feature channel weighted averaging unit, used to weight the temporal attention features of the speech sample; and a temporal encoding unit, used to encode the weighted temporal attention features to obtain enhanced temporal features, i.e., time-frequency features.

[0039] Preferably, the emotion classifier module specifically includes: an emotion feature mapping unit, used to perform emotion feature space mapping on the high-level features of the speech sample; and an emotion feature classification unit, used to classify and predict the emotion features of the speech sample.

[0040] The beneficial effects of this invention are as follows: First, the log-Mel spectrum features obtained by the sample feature extraction module are more robust. Then, the time-frequency attention network and encoder obtain features that are independent of abnormal noise (such as background noise, speaker identity, language, etc.) but related to emotion. Finally, the emotion classifier module is used to further enhance the emotion discrimination and noise-independent generalization of the obtained features. This invention has better recognition effect and higher recognition rate. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0042] Figure 2 This diagram illustrates the comparison of experimental results between the present invention and other methods on the IEMOCAP dataset.

[0043] Figure 3 This diagram illustrates the comparison of experimental results between the present invention and other methods on the ABC dataset.

[0044] Figure 4 This is the normalized confusion matrix of the present invention on the IEMOCAP dataset.

[0045] Figure 5 This is the normalized confusion matrix of the present invention on the ABC dataset. Detailed Implementation

[0046] like Figure 1 As shown, a speech emotion recognition method based on attention time-frequency networks includes the following steps:

[0047] (1) Obtain emotional speech samples, perform short-time discrete Fourier transform to obtain spectral features, and use them as network input.

[0048] This step specifically includes: (1-1) Preprocessing the denoised speech signal by pre-emphasis, framing, and windowing; (1-2) Performing short-time discrete Fourier transform on the preprocessed speech signal to obtain the spectrum; (2-3) Scaling the spectrum by Mel scale through multiple Mel filter banks and then removing the logarithm to obtain the log-Mel spectrum features as the input of the network.

[0049] In this embodiment, framing, windowing, short-time discrete Fourier transform, and Mel filter bank all use the Librosa open-source speech signal processing library. This library uses a Hamming window with a window length of 25ms, an inter-frame overlap rate of 50%, 512 FFT points, and 80 Mel filters.

[0050] (2) Input the log-Mel spectrum features of the training samples into the attention-based frequency domain encoder to obtain weighted frequency domain speech emotion features.

[0051] This step specifically includes: (2-1) inputting training samples into the frequency domain attention network; (2-2) the frequency domain attention network contains a convolutional module and a broadcast multiplication module, wherein each convolutional module contains three operations: 1D convolution, batch normalization, and ReLU, and the broadcast multiplication module contains one broadcast multiplication operation; (2-3) inputting the frequency domain attention a f The broadcast product of the frequency domain feature x is obtained (2-4) Then The weighted frequency domain feature x′ is obtained by adding it to the original frequency domain feature x through each channel and averaging the result.

[0052] In this embodiment, the convolution sizes are (1, 5, 8), the frequency domain attention convolution size is (1, 5), and the number of channels is 8; the frequency domain encoder has 2 convolutions, the tape measure sizes are (1, 5), and the number of channels is 32 and 8.

[0053] (3) Weighted frequency domain speech emotion features are input to the attention-based time domain encoder to obtain weighted time domain speech emotion features, i.e., time frequency speech emotion features.

[0054] This step specifically includes: (3-1) inputting training samples into the temporal attention network; (3-2) the temporal attention network contains a convolutional module and a broadcast multiplication module, wherein each convolutional module contains three operations: 1D convolution, batch normalization, and ReLU, and the broadcast multiplication module contains one broadcast multiplication operation; (3-3) inputting the temporal attention a t The broadcast product of the weighted frequency domain feature x′ is obtained. (3-4) Then The weighted frequency domain feature x′ is added to the channel-wise feature and then averaged and input to the time domain encoder to obtain the enhanced time domain feature, namely the weighted time-frequency feature x″.

[0055] In this embodiment, the convolution sizes are (1, 8, 8), the convolution size of the temporal attention is (1, 5), and the number of channels is 8; the temporal encoder is a bidirectional LSTM with 512 hidden nodes.

[0056] (4) The time-frequency features are classified using an emotion classifier.

[0057] This step specifically includes: (4-1) High-level features G of the source domain s Emotion classification is performed by an emotion discriminator; (4-2) The emotion discriminator is a two-layer fully connected network with hidden node dimensions of 512 and emotion category number c.

[0058] In this embodiment, the emotional category c is set according to the specific database: the emotional category c for IEMOCAP is 4, and the emotional category c for ABC is 6.

[0059] (5) Jointly train the frequency domain attention network, frequency domain encoder, time domain attention network, time domain encoder, and sentiment classifier to obtain the optimal model parameters.

[0060] In this embodiment, the frequency domain attention network, frequency domain encoder, temporal domain attention network, temporal encoder, and sentiment classifier all employ the stochastic gradient descent method.

[0061] (6) Input the emotional speech sample to be identified into the trained frequency domain attention network, frequency domain encoder, time domain attention network, time domain encoder, and emotion classifier to obtain the identified speech emotion.

[0062] The neural network was trained and then tested. The test results are as follows:

[0063] To verify the effectiveness and necessity of the attention-based time-frequency network of this invention, experiments were conducted on the IEMOCAP and ABC datasets. Weighted average recall (WAR) and unweighted average recall (UAR) were selected as evaluation metrics. The results are as follows: Figure 2 and Figure 3 As shown, the attention time-frequency network of this invention achieved the best results on both datasets. To avoid the problem that the recognition rate cannot be objectively evaluated due to the imbalance of the number of emotion samples in the datasets, we calculated the normalized confusion matrix for the results of the attention time-frequency network on the two datasets, as shown below. Figure 4 and Figure 5 As shown.

[0064] This embodiment also provides a speech emotion recognition system based on an attention time-frequency network, including:

[0065] The feature extraction module is used to extract the log-Mel spectrum features of speech samples;

[0066] The frequency domain attention module is used to compute the frequency domain attention of Mel features;

[0067] The frequency domain encoder module is used to perform frequency domain encoding on the Mel features after frequency domain attention enhancement;

[0068] The temporal attention module is used to compute temporal attention for Mel features;

[0069] The time-domain encoder module is used to perform time-domain encoding on the frequency-domain features after time-domain attention enhancement;

[0070] The sentiment classifier module is used to classify sentiment categories based on time-frequency features.

[0071] The attention time-frequency network training module is used to establish a time-frequency joint learning network composed of a frequency domain attention network, a frequency domain encoder network, a time domain attention network, a time domain encoder network, and a sentiment classification network. The corresponding features of each path are used as input to train the attention time-frequency network.

[0072] The speech emotion recognition module is used to input the speech sample to be recognized into a trained attention time-frequency network to obtain the recognized speech emotion.

[0073] The feature extraction module specifically includes:

[0074] The framing and windowing unit is used to perform framing and windowing operations on the clean speech signal;

[0075] The spectrum extraction unit is used to extract the spectrum of the framed and windowed speech signal by performing a short-time discrete Fourier transform.

[0076] The log-Mel spectrum extraction unit is used to perform Mel-scale filtering and logarithmic operations on the extracted spectrum to obtain log-Mel spectrum features.

[0077] The frequency domain attention module specifically includes:

[0078] A frequency domain attention calculation unit is used to obtain the frequency domain attention of the Mel features of the speech sample;

[0079] The frequency domain attention broadcast product unit is used to obtain the frequency domain attention features of the speech sample.

[0080] The frequency domain encoder module specifically includes:

[0081] A frequency domain feature channel weighted averaging unit is used to perform frequency domain attention weighted features on the speech samples;

[0082] The frequency domain coding unit is used to perform frequency domain feature encoding on the frequency domain attention-weighted features to obtain enhanced frequency domain features.

[0083] The temporal attention module specifically includes:

[0084] A temporal attention calculation unit is used to obtain the temporal attention of the Mel features of the speech sample;

[0085] The temporal attention broadcast product unit is used to obtain the temporal attention features of the speech sample.

[0086] The time-domain encoder module specifically includes:

[0087] A time-domain feature channel weighted averaging unit is used to perform time-domain attention-weighted features on the speech samples;

[0088] The temporal coding unit is used to encode the temporal attention-weighted features to obtain enhanced temporal features, namely, time-frequency features.

[0089] The sentiment classifier module specifically includes:

[0090] The emotion feature mapping unit is used to perform emotion feature space mapping on the high-level features of the speech sample;

[0091] The emotion feature classification unit is used to classify and predict the emotion features of the speech samples.

Claims

1. A speech emotion recognition method based on attention time-frequency networks, characterized in that, Includes the following steps: (1) Obtain several emotional speech samples, perform short-time discrete Fourier transform, and then pass them through a Mel filter bank. Take the logarithm to obtain the log-Mel spectrum features, which are used as the input of the network. (2) Input the log-Mel spectrum features of the training samples into the attention-based frequency domain encoder to obtain weighted frequency domain speech emotion features; specifically including the following steps: (2-1) Input the log-Mel spectrum features of the training samples into the frequency domain attention network to obtain the frequency domain attention a. f ; (2-2) Frequency domain attention a f After multiplying the frequency domain feature x with the original frequency domain feature x by the broadcast product, the average value of the result is obtained as the weighted frequency domain feature x′. (2-3) The frequency domain attention network contains a convolution module, a product module and a summation module. Each convolution module contains a convolution module operation, each product module contains a broadcast product operation, and each summation module contains an addition operation. (3) Input the weighted frequency domain speech emotion features into the attention-based time domain encoder to obtain the weighted time domain speech emotion features, i.e., the time frequency speech emotion features; (4) The time-frequency speech emotion features are classified using an emotion classifier; (5) Jointly train the attention-based frequency domain encoder, the attention-based time domain encoder, and the sentiment classifier to obtain the optimal model parameters; (6) Input the emotional speech sample to be identified into the trained attention-based frequency domain encoder, attention-based time domain encoder and emotion classifier to obtain the identified speech emotion.

2. The speech emotion recognition method based on attention time-frequency network as described in claim 1, characterized in that, In step (1), several emotional speech samples are acquired, and after performing a short-time discrete Fourier transform, they are passed through a Mel filter bank, and the logarithmic Mel spectral features are obtained as the input to the network. The specific steps include the following: (1-1) Perform pre-emphasis, framing, and windowing preprocessing on the denoised speech signal; (1-2) The spectrum is obtained by performing a short-time discrete Fourier transform on the preprocessed speech signal; (1-3) The spectrum is scaled by Mel scale through multiple Mel filter banks and then logarithmized to obtain the log-Mel spectrum features, which are used as the input of the network.

3. The speech emotion recognition method based on attention time-frequency network as described in claim 1, characterized in that, In step (3), the weighted frequency domain speech emotion features are input into the attention-based time domain encoder to obtain the weighted time domain speech emotion features, i.e., the time frequency speech emotion features. Specifically, this includes the following steps: (3-1) Input the log-Mel spectrum features of the training samples into the temporal attention network to obtain the temporal attention a. t ; (3-2) Apply temporal attention a t After multiplying the weighted frequency domain feature x′ by the broadcast product, and then weighting the weighted frequency domain feature x′ by channel, the average value is used to obtain the weighted time domain feature, i.e., the weighted time-frequency feature x″. (3-3) The temporal attention network contains a convolution module, a product module and a summation module. Each convolution module contains a convolution module operation, each product module contains a broadcast product operation, and each summation module contains an addition operation.

4. The speech emotion recognition method based on attention time-frequency network as described in claim 1, characterized in that, In step (4), the classification of time-frequency features by the emotion discriminator specifically includes the following steps: (4-1) The weighted time-frequency feature x″ is used for sentiment classification by a sentiment discriminator; (4-2) The sentiment discriminator is a two-layer fully connected network with hidden node dimensions of 512 and sentiment category number c.

5. The speech emotion recognition system based on the attention time-frequency network as described in claim 1, characterized in that, include: The feature extraction module is used to extract the log-Mel spectrum features of speech samples; The frequency domain attention module is used to compute the frequency domain attention of Mel features; The frequency domain encoder module is used to perform frequency domain encoding on the Mel features after frequency domain attention enhancement; The temporal attention module is used to compute temporal attention for Mel features; The time-domain encoder module is used to perform time-domain encoding on the frequency-domain features after time-domain attention enhancement; The sentiment classifier module is used to classify sentiment categories based on time-frequency features. The attention time-frequency network training module is used to establish a time-frequency joint learning network composed of a frequency domain attention network, a frequency domain encoder network, a time domain attention network, a time domain encoder network, and a sentiment classification network. The corresponding features of each path are used as input to train the attention time-frequency network. The speech emotion recognition module is used to input the speech sample to be recognized into a trained attention time-frequency network to obtain the recognized speech emotion.

6. The speech emotion recognition system based on the attention time-frequency network as described in claim 5, characterized in that, The feature extraction module specifically includes: a framing and windowing unit, used to perform framing and windowing operations on the clean speech signal; a spectrum extraction unit, used to perform short-time discrete Fourier transform on the framed and windowed speech signal to extract the spectrum; and a log-Mel spectrum extraction unit, used to perform Mel-scale filtering and logarithmic operation on the extracted spectrum to obtain log-Mel spectrum features.

7. The speech emotion recognition system based on the attention time-frequency network as described in claim 5, characterized in that, The frequency domain attention module specifically includes: a frequency domain attention calculation unit, used to obtain the frequency domain attention of the Mel features of the speech sample; and a frequency domain attention broadcast product unit, used to obtain the frequency domain attention features of the speech sample. The frequency domain encoder module specifically includes: a frequency domain feature channel weighted averaging unit, used to obtain the frequency domain attention weighted features of the speech sample; and a frequency domain encoding unit, used to perform frequency domain feature encoding on the frequency domain attention weighted features to obtain enhanced frequency domain features.

8. The speech emotion recognition system based on the attention time-frequency network as described in claim 5, characterized in that, The temporal attention module specifically includes: a temporal attention calculation unit, used to obtain the temporal attention of the Mel features of the speech sample; and a temporal attention broadcast product unit, used to obtain the temporal attention features of the speech sample. The temporal encoder module specifically includes: a temporal feature channel weighted averaging unit, used to obtain the temporal attention weighted features of the speech sample; and a temporal encoding unit, used to perform temporal feature encoding on the temporal attention weighted features to obtain enhanced temporal features, i.e., time-frequency features.

9. The speech emotion recognition system based on the attention time-frequency network as described in claim 5, characterized in that, The emotion classifier module specifically includes: an emotion feature mapping unit, used to perform emotion feature space mapping on the high-level features of the speech sample; and an emotion feature classification unit, used to classify and predict the emotion features of the speech sample.

Images