A Speech Emotion Recognition Method and Device Based on an Improved Attention Mechanism

Abstract

The present invention provides a method and device for speech emotion recognition based on an improved attention mechanism. The method comprises: performing data preprocessing on collected audio signal samples; extracting acoustic features from the collected audio signal samples to obtain a spectral feature map; constructing a CNN-BGRU deep learning network combining a convolutional neural network with a bidirectional gated recurrent unit (BGRU) network to extract deeper semantic and contextual information from the spectral feature map; constructing an improved MSK attention mechanism module to further process the feature information obtained by the CNN-BGRU deep learning network; and outputting the speech emotion recognition results processed by the improved MSK attention mechanism module. The method effectively improves the detection accuracy of speech emotion recognition and significantly reduces recognition time without increasing the number of model parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of speech emotion recognition, and in particular to a speech emotion recognition method and device based on an improved attention mechanism. Background Art

[0002] In recent years, with the development of deep learning and artificial intelligence technology, human-computer interaction systems have become more and more intelligent and humanized. Voice interaction is an indispensable and important component. Voice not only carries voice information, but also contains rich emotional information. Voice emotion recognition technology has been widely used in various fields such as in-vehicle driving systems, mobile phone voice assistants, medical care, education, etc.

[0003] Speech emotion recognition systems generally consist of three parts: speech signal acquisition, emotion feature extraction, and emotion recognition. The system includes the following elements: acoustic feature parameters, emotion classification models, and emotion corpora. Common acoustic features include amplitude, pitch frequency, MFCCs, and formants. Traditional machine learning methods, such as Gaussian mixture models, hidden Markov models, and support vector machines, often combine deep neural networks (DNNs) with hidden Markov models (HMMs) to form DNN-HMN acoustic models. However, these models train their constituent units independently, without considering the impact of relationships between these units on model training performance. Previous research has widely used convolutional neural networks (CNNs) to classify extracted features. Inspired by image recognition tasks in computer vision, convolutional neural networks have also made significant progress in speech emotion recognition. Some researchers have applied CNNs to spectrograms to extract features for emotion classification, achieving state-of-the-art recognition results on the popular IMEOCAP dataset. Some researchers have also adopted a composite framework consisting of three 2D CNN layers, two fully connected layers FC1 and FC2, and a bidirectional recurrent neural network layer. However, this easily makes the model network computationally too large, leading to the gradient explosion problem and ultimately significantly affecting the accuracy of speech emotion recognition.

[0004] In summary, existing technologies face the following challenges: existing speech emotion recognition models fail to fully exploit the speech emotion information frame, preventing some hidden state information from being extracted by the neural network, ultimately leading to the loss of emotion information frames. Furthermore, the attention neural network models constructed using existing speech emotion recognition methods involve a large amount of redundant computation for similar emotions, resulting in excessive computational complexity and time-consuming processing, ultimately impacting speech emotion recognition accuracy. Summary of the Invention

[0005] In response to the defects in the prior art, the purpose of the present invention is to provide a method and device for speech emotion recognition based on an improved attention mechanism. The method solves the problems that the current neural network used for speech emotion recognition does not fully mine the emotion information frames, and there are a large number of redundant calculations for similar emotions, which takes too long and affects the accuracy of speech emotion recognition.

[0006] In order to solve the above problems, the technical solution of the present invention is:

[0007] A speech emotion recognition method based on an improved attention mechanism includes the following steps:

[0008] Performing data preprocessing on the collected audio signal samples;

[0009] Extracting acoustic features from the collected audio signal samples to obtain a spectrum feature graph;

[0010] Construct a CNN-BGRU deep learning network that combines a convolutional neural network with a bidirectional gated recurrent unit network to extract deeper semantic and contextual information from the spectral feature map;

[0011] Construct an MSK improved attention mechanism module to further process the feature information obtained through the CNN-BGRU deep learning network;

[0012] Output the speech emotion recognition results processed by the MSK improved attention mechanism module.

[0013] Preferably, in the step of performing data preprocessing on the collected audio signal samples, the preprocessing of the speech signal includes speech sampling and quantization, speech framing, and signal windowing. The speech signal is framed, and the speech signal framing requires a windowing operation. The window function formula is:

[0014] Preferably, in the step of extracting acoustic features from the collected audio signal samples to obtain a spectrum feature map, the acoustic features include Mel-frequency cepstral coefficients, spectrum center, spectrum entropy, chroma features, spectrum extension, and chroma feature standard deviation.

[0015] Preferably, the extraction of the Mel-frequency cepstral coefficient features comprises the following steps:

[0016] The original one-dimensional emotional speech signal was resampled to a 16kHz mono format, and a Hanning window with a frame shift of 10ms and a duration of 25ms was used to perform a short-time Fourier transform to calculate the spectrogram of the entire speech signal.

[0017] The spectrogram is mapped to a 64-order Mel filter bank to calculate the Mel spectrum and take its logarithm to obtain a stable logarithmic Mel spectrum. The Mel scale describes the nonlinear characteristics of the human ear frequency. Its relationship with frequency can be approximately expressed as follows:

[0018] Preferably, the step of constructing a CNN-BGRU deep learning network combining a convolutional neural network with a bidirectional gated recurrent unit network to extract deeper semantic information and contextual information from the spectral feature graph specifically includes:

[0019] In order to obtain the long-term dynamic information of a speech, BGRU is used to model the temporal dynamic information of the feature sequences learned by CNN on different speech spectrum segments;

[0020] The BGRU model consists of an update gate and a reset gate, and the construction formula is as follows:

[0021] r=σ(J r x t +T r h t-1 )

[0022] z=σ(J z x t +T z h t-1 )

[0023] g=tanh(J g x t +T g rh t-1 )

[0024] h t =(1-z)h t-1 +zg

[0025] Among them, z is the update gate, r is the reset gate, σ is the activation function representing the activation rate of neurons in the network, J r and J z represents the weight matrix, g is the candidate hidden state in the network, T is used to represent the current moment, and h t-1 is the output at the previous moment, h t is the output at the current moment.

[0026] Preferably, the CNN-BGRU deep learning network specifically includes:

[0027] The three-dimensional data output by the CNN feature extraction network is globally pooled in the frequency and time dimensions to form one-dimensional data in the channel dimension. The channel is then reduced and then increased in dimension using a two-layer full connection. The channel attention score is then calculated using the Sigmoid function and multiplied with the original CNN output feature to achieve feature attention in the channel dimension.

[0028] The BGRU network is used to capture the refined emotional features of the spectrogram in terms of space and channels, and the spectrogram feature map passed in by the CNN layer is further processed to obtain deeper emotional semantic information and contextual information.

[0029] Preferably, in the step of constructing the MSK improved attention mechanism module and further processing the feature information obtained through the CNN-BGRU deep learning network, the MSK improved attention mechanism module specifically includes: initially splitting a continuous set of q(x), matching a set of similar keys within a continuous time and performing clustered storage processing.

[0030] Preferably, the attention value formula after fusion of MSK is:

[0031]

[0032] w=Softmax((q(x) T k i ) i∈P )

[0033] Attention(q, K, V) = ∑ i∈P w i v i

[0034] Select m similar keys corresponding to the current set of queries, use the inner product of each query and key in the group as the similarity, then perform softmax normalization function to obtain the one with the highest similarity, and then calculate the attention score by weighted summation with the corresponding value.

[0035] Furthermore, the present invention also provides a speech emotion recognition device based on an improved attention mechanism, characterized in that the device includes a processor and a memory for storing executable instructions of the processor, and the processor is configured to execute the speech emotion recognition method based on the improved attention mechanism as described above by executing the executable instructions.

[0036] Compared with the existing technology, the speech emotion recognition method based on the improved attention mechanism of the present invention introduces a novel MSK module on the basis of the traditional attention mechanism, which can greatly reduce redundant attention score calculations to reduce model time consumption, effectively improve the detection accuracy of speech emotion recognition, and greatly reduce the time spent on recognition without increasing the number of parameters of the model. It solves the problems of insufficient mining of emotion information frames in the current neural network used for speech emotion recognition, and a large number of redundant calculations for similar emotions, which takes too long and affects the accuracy of speech emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0038] Figure 1 A flowchart of a speech emotion recognition method based on an improved attention mechanism provided by an embodiment of the present invention;

[0039] Figure 2 A detailed flow chart of the speech emotion recognition method based on the improved attention mechanism provided by an embodiment of the present invention;

[0040] Figure 3 An audio signal processing flow chart of the speech emotion recognition method based on the improved attention mechanism provided by an embodiment of the present invention;

[0041] Figure 4 This is a confusion matrix result diagram of the speech emotion recognition method based on the improved attention mechanism of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0043] Dataset selection and experimental environment introduction: The CREMA-D dataset is a speech dataset of 7442 clips recorded by 91 actors from different races and ethnicities.

[0044] The hardware environment of the experiment is an Intel Core i7-5820K CPU, the operating system is Windows 10, the programming language is Python, and the experimental platform is Pycharm.

[0045] Specifically, the embodiment of the present invention provides a speech emotion recognition method based on an improved attention mechanism, such as Figure 1and Figure 2 As shown, the method includes the following steps:

[0046] S1: perform data preprocessing on the collected audio signal samples;

[0047] First, check whether the format of all audio files is legal. The specific method is to check the suffix of the file name. If the suffix is in the legal list (such as '.mp3', '.wav'), the format meets the requirements. Otherwise, use the open source Python library to convert the audio files to the '.wav' format.

[0048] Specifically, the preprocessing of the speech signal is divided into three parts: speech sampling and quantization, speech framing, and signal windowing. When the characteristics of the speech signal change little within a short time range, it can be considered a steady-state signal. The speech signal is framed, and the frame length is generally 10 to 30 ms. Speech signal framing usually requires a windowing operation. The window function can reduce the impact of frequency domain energy leakage caused by truncation. The Hamming window formula is as follows:

[0049]

[0050] S2: extracting acoustic features from the collected audio signal samples;

[0051] The acoustic features include Mel-frequency cepstral coefficients (MFCC), spectrum center, spectrum entropy, chroma features, spectrum spread, chroma feature standard deviation, etc. In this embodiment, Python's librosa toolkit is used to extract Mel-frequency cepstral coefficient related features.

[0052] Specifically, if Figure 3 As shown, the extraction process of the MFCC related features is as follows:

[0053] The original one-dimensional emotional speech signal is resampled to a 16kHz monophonic format. A short-time Fourier transform is then performed using a Hanning window with a frame shift of 10ms and a duration of 25ms. The spectrogram of the entire speech signal is calculated. The spectrogram is then mapped to a 64-order Mel filter bank to calculate the Mel spectrum and take its logarithm, resulting in a stable logarithmic Mel spectrum. MFCCs are cepstral parameters extracted in the Mel scale frequency domain. The Mel scale describes the nonlinear characteristics of the human ear's frequency response. Its relationship to frequency can be approximately expressed as follows:

[0054] S3: Construct a CNN-BGRU deep learning network that combines a convolutional neural network with a bidirectional gated recurrent unit network;

[0055] Specifically, the CNN-BGRU deep learning network is used to extract the emotional semantics and related emotional features of the spectrogram. In order to obtain the long-term dynamic information of a sentence of speech, a bidirectional gated recurrent unit (BGRU) based on the attention mechanism is used to model the temporal dynamic information of the 128-D feature sequence learned by CNN on different speech spectrum segments, and output the emotion recognition result of the entire speech sample. The GRU is mainly composed of two parts: the reset gate and the update gate. The update gate is used to update the degree to which the neural unit remembers valid information at the current moment. The larger the value of the update gate, the more valid information is remembered. The following are the operational expressions of the reset gate and the update gate:

[0056] r=σ(J r x t +T r h t-1 )

[0057] z=σ(J z x t +T z h t-1 )

[0058] g=tanh(J g x t +T g rh t-1 )

[0059] h t =(1-z)h t-1 +zg

[0060] In the above formula, z is the update gate, r is the reset gate, and σ is the activation function representing the activation rate of neurons in the network. r and J z Represents the weight matrix corresponding to the reset gate and update gate output at the previous moment. g is the candidate hidden state in the network, which is usually generated between the reset gate and update gate state exchange. Thus J g Represents the weight matrix corresponding to the candidate hidden state at the previous moment. The current moment is represented by T, so T z , T r and T g Respectively represent the weight matrices corresponding to the update gate, reset gate and hidden state at the current moment, h t-1 is the output at the previous moment, h t is the output at the current moment.

[0061] Furthermore, the CNN-BGRU deep learning network in step S3 specifically includes:

[0062] (1) The three-dimensional data output by the CNN feature extraction network is globally pooled in the frequency and time dimensions to form one-dimensional data in the channel dimension. The channel is then reduced and then increased in dimension through a two-layer full connection. The channel attention score is calculated using the Sigmoid function and multiplied with the original CNN output feature to achieve feature attention in the channel dimension.

[0063] (2) The BGRU network is used to capture the refined emotional features of the spectrogram in terms of space and channels, and the spectrogram feature map passed by the CNN layer is further processed to obtain deeper emotional semantic information and contextual information. Based on the GRU model's ability to process variable-length speech features, the ends of speech segments of different lengths are padded with zeros to make them the same length. When training the parameters of the bidirectional GRU, only the actual valid speech data is used in the calculation, and the data padded with zeros is not used in the calculation.

[0064] S4: Build the MSK improved attention mechanism module to further process the feature information obtained through the CNN-BGRU deep learning network;

[0065] Specifically, similar emotional information is first pre-processed through MSK to reduce a large number of redundant attention score calculations, filter irrelevant features, reduce the influence of emotional insensitivity factors, and assist the deep learning network in capturing refined features in terms of space and channels.

[0066] Soft attention involves three key elements: query, key, and value. These represent the query value, keyword, and weight corresponding to the input speech feature, respectively. Correlation is calculated by aligning the query and key as closely as possible. The resulting correlation value is then used to determine the distribution vector of weight coefficients for each key-value pair.

[0067] First, the additive model is used to calculate the similarity between query and key.

[0068] s(k i ,q)=v T tanh(Wk i +Uq)

[0069] Where U, W, and v are learnable parameters in the network, tanh is the activation function, and s(k_i,q) is the attention scoring function, which represents the similarity score between the i-th key and the query value query.

[0070] Calculate the attention similarity score: s(k i ,q)=v T o(Wk i+Uq+p1)+p2, where p_1 and p_2 are biases, σ is the activation function, and W and U are parameters learned by the network.

[0071] When the input feature parameters query vector and key vector are given, the softmax function is first used to find the weight of the i position vector.

[0072]

[0073] After calculating all weight coefficients, perform weighted summation on the value to get the corresponding attention value:

[0074]

[0075] If the input sample contains a large number of consecutive speech emotion frames of the same category or with high similarity within a certain period of time, it will be very time-consuming to re-match similar keys for each query and recalculate the similarity between them, which will significantly affect the recognition efficiency of the entire network.

[0076] This paper improves on the traditional attention mechanism and proposes an improved attention mechanism module that integrates MSK. Initially, a continuous set of q(x) is split, and a group of similar keys are matched within a continuous period of time and clustered and stored. The attention value formula after integrating MSK is:

[0077]

[0078] w=Softmax((q(x) T k i ) i∈P )

[0079] Attention(q, K, V) = ∑ i∈P w i v i

[0080] First, m similar keys corresponding to the current set of queries are selected, and the inner product of each query and key in the group is used as the similarity. Then, the softmax normalization function is performed to obtain the one with the highest similarity, and the attention score is calculated by weighted summation with the corresponding value.

[0081] S5: Outputs the speech emotion recognition results processed by the MSK improved attention mechanism module.

[0082] Specifically, if Figure 4 As shown in the figure, the confusion matrix result of speech emotion recognition processed by the improved attention mechanism module is output.

[0083] The comparative analysis between the model of the present invention and other models is as follows:

[0084] The MSK Attention model in this example is integrated with the CNN and BGRU neural networks to build an overall model. Accuracy, recall, and F1 score are used as evaluation indicators. The following five model comparison experiments are set up on the CREMA-D dataset:

[0085] 1. LSTM model.

[0086] We used the original MFCC feature extraction method to extract MFCC emotion features, followed by pre-emphasis and framing, and finally second-order differencing. We then used a traditional LSTM model for emotion classification and recognition, followed by a fully connected layer and a softmax layer. This is the most commonly used method for speech emotion recognition. We used this experiment as a benchmark for optimization and comparative experiments around this baseline model.

[0087] 2. CNN+BLSTM model.

[0088] First, CNN is used to obtain the emotional features in the spectrogram, so that the emotional semantics can be better captured later. The number of CNN layers is set to 2, and the convolution kernel size is 3×3.

[0089] 3. CNN+GRU model.

[0090] The GRU network is used to replace the LSTM network in the second set of experiments. The purpose of the experiment is to verify whether the GRU network can improve the accuracy of emotion recognition and reach convergence with fewer iterations than the LSTM network.

[0091] 4. CNN+BLSTM+Attention model.

[0092] Based on the second model, the attention mechanism is incorporated to perform importance scoring on the output speech features at each moment in the bidirectional LSTM, hoping to highlight the importance of the emotional information frame. The state vector of the BLSTM at the last moment is input into a two-layer fully connected layer with 256 nodes.

[0093] 5. CNN+BGRU+Attention model.

[0094] This set of experiments uses CNN and BGRU networks while incorporating attention mechanism modules, hoping that the network can focus more on learning emotional information during training, distinguish the importance of different speech frames, assign different weights, and finally perform weighted output.

[0095] Table 1 below shows the speech emotion recognition results of different schemes on the CREMA-D dataset

[0096] Model Accuracy Recall F1 LSTM 61.23 57.45 59.20 CNN+LSTM 72.14 78.64 70.13 CNN+GRU 82.56 77.62 79.86 CNN+BLSTM+MSK Attention 84.03 82.37 83.52 CNN+BGRU+MSK Attention 86.78 81.67 82.54

[0097] Table 1

[0098] The CNN structure consists of two convolutional layers. The first convolutional layer has a kernel size of 2*2, a stride of 1, and 10 kernels. The second convolutional layer has a kernel size of 2*2, a stride of 1, and 20 kernels. Each convolutional layer is followed by a maximum pooling layer of 2*2 with a stride of 2. Finally, two fully connected layers with 128 neurons are added, and a batch normalization layer is added between the fully connected layers. In this experiment, the GRU is set to two layers, with 256 neurons in each layer and a random inactivation rate (dropout) of 0.3. The experimental parameter settings of the BGRU are the same as those of the GRU, except that a reverse GRU layer is added after each forward GRU layer.

[0099] The second and third experiments in Table 1 show that the combination of the GRU model and CNN network outperforms the traditional LSTM model and CNN combination. This is because the GRU network structure is relatively simple, with only update and reset gates, and the hidden state is directly passed to the next unit. Therefore, it has fewer parameters and is easier to train to converge, thus improving performance. Incorporating the attention mechanism into both sets of experiments significantly improved accuracy. Because the emotional features in speech signals are contextually continuous, the attention mechanism can more closely capture the emotional information frames within the speech sample, thereby improving the recognition accuracy of the speech emotion recognition model. The fusion of the BGRU network and the attention mechanism performed even better, achieving a 2.75% increase in accuracy over the BLSTM and attention fusion model. This is because the BGRU converges in fewer iterations, significantly improving the computational efficiency of the overall network.

[0100] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A speech emotion recognition method based on an improved attention mechanism, characterized in that: The method comprises the following steps: Performing data preprocessing on the collected audio signal samples; Extracting acoustic features from the collected audio signal samples to obtain a spectrum feature graph; Construct a CNN-BGRU deep learning network that combines a convolutional neural network with a bidirectional gated recurrent unit network to extract deeper semantic and contextual information from the spectral feature map; Constructing an MSK improved attention mechanism module to further process the feature information obtained through the CNN-BGRU deep learning network. The construction of the MSK improved attention mechanism module specifically includes: initially splitting a continuous set of q(x), matching a set of similar keys within a continuous period of time and performing clustered storage processing; the attention value formula after integrating MSK is: w=Softmax((q(x) T k i ) i∈P ) Attention(q,K,V)=∑ i∈P w i v i Select m similar keys corresponding to the current set of queries, use the inner product of each query and key in the group as the similarity score, then perform softmax normalization to obtain the weight distribution, highlight the key-value pairs with the highest similarity through weighted summation, and finally calculate the attention score; Output the speech emotion recognition results processed by the MSK improved attention mechanism module.

2. The speech emotion recognition method based on the improved attention mechanism according to claim 1 is characterized in that In the step of performing data preprocessing on the collected audio signal samples, the preprocessing of the audio signal includes voice sampling and quantization, voice framing, and signal windowing. The voice signal is framed, and the voice signal framing requires a windowing operation. The window function formula is:

3. The speech emotion recognition method based on the improved attention mechanism according to claim 1 is characterized in that In the step of extracting acoustic features from the collected audio signal samples to obtain a spectrum feature map, the acoustic features include Mel-frequency cepstral coefficients, spectrum center, spectrum entropy, chroma features, spectrum expansion, and chroma feature standard deviation.

4. The speech emotion recognition method based on the improved attention mechanism according to claim 3 is characterized in that Extracting the Mel-frequency cepstral coefficient features includes the following steps: The original one-dimensional emotional speech signal was resampled to a 16kHz mono format, and a Hanning window with a frame shift of 10ms and a duration of 25ms was used to perform a short-time Fourier transform to calculate the spectrogram of the entire speech signal. The spectrogram is mapped to a 64-order Mel filter bank to calculate the Mel spectrum and take its logarithm to obtain a stable logarithmic Mel spectrum. The Mel scale describes the nonlinear characteristics of the human ear frequency. Its relationship with frequency can be approximately expressed as follows:

5. The speech emotion recognition method based on the improved attention mechanism according to claim 1 is characterized in that The steps of constructing a CNN-BGRU deep learning network combining a convolutional neural network with a bidirectional gated recurrent unit network to extract deeper semantic information and contextual information from the spectral feature graph specifically include: In order to obtain the long-term dynamic information of a speech, BGRU is used to model the temporal dynamic information of the feature sequences learned by CNN on different speech spectrum segments; The BGRU model consists of an update gate and a reset gate, and the construction formula is as follows: r=σ(J r x t +T r h t-1 ) z=σ(J z x t +T z h t-1 ) g=tanh(J g x t +T g rh t-1 ) h t =(1-z)h t-1 +zg Among them, z is the update gate, r is the reset gate, σ is the activation function representing the activation rate of neurons in the network, J r and J z represents the weight matrix, g is the candidate hidden state in the network, t is used to represent the current moment, and h t-1 is the output at the previous moment, h t is the output at the current moment.

6. The speech emotion recognition method based on the improved attention mechanism according to claim 5 is characterized in that: The CNN-BGRU deep learning network specifically includes: The three-dimensional data output by the CNN feature extraction network is globally pooled in the frequency and time dimensions to form one-dimensional data in the channel dimension. The channel is then reduced and then increased in dimension using a two-layer full connection. The channel attention score is then calculated using the Sigmoid function and multiplied with the original CNN output feature to achieve feature attention in the channel dimension. The BGRU network is used to capture the refined emotional features of the spectrogram in terms of space and channels, and the spectrogram feature map passed in by the CNN layer is further processed to obtain deeper emotional semantic information and contextual information.

7. A speech emotion recognition device based on an improved attention mechanism, characterized in that: The device includes a processor and a memory for storing executable instructions of the processor, wherein the processor is configured to perform the speech emotion recognition method based on the improved attention mechanism according to any one of claims 1 to 6 by executing the executable instructions.

Images