A speech emotion recognition method based on time-frequency attention mechanism

Abstract

The application discloses a speech emotion recognition method based on a time-frequency attention mechanism and belongs to the technical field of speech emotion recognition. The log-Mel spectrogram feature is extracted, the log-Mel spectrogram feature is lengthened by using a cyclic padding method, and the lengthened log-Mel spectrogram feature is input into a speech emotion recognition model, which comprises sequentially connected time-frequency convolution modules, a time-frequency attention module, a multilayer convolution network and a full connection layer; the time-frequency convolution module captures time-domain, frequency-domain change information and time-frequency features, and the time-frequency attention module generates a time-frequency weighted feature map; then, deep emotion features are learned through the multilayer convolution network; and different speech emotions are classified by using a Softmax classifier. The method is simple in steps and can more accurately recognize emotions in speech.

Description

Technical Field

[0001] This invention relates to a speech emotion recognition method based on a time-frequency attention mechanism, belonging to the field of speech emotion recognition technology. Background Technology

[0002] Speech emotion recognition has significant application value in human-computer interaction. Traditional speech emotion recognition methods are based on acoustic features and machine learning classification models to identify and classify external emotions. Acoustic features are mostly extracted from raw audio recordings, including low-level descriptors (LLDs) and high-level statistical features (HSFs). Building upon this, to further refine the extracted acoustic features for emotion recognition, various machine learning classification models, including Markov models, Gaussian mixture models, and decision trees, are often employed. However, acoustic features cannot effectively represent linguistic emotional information; that is, when selecting frequency domain features, time-related emotional information is often ignored, and conversely, when selecting time-domain features, frequency-related information is often overlooked.

[0003] Since emotional information in speech is distributed in both the time and frequency domains, spectrograms, being time-frequency maps, can preserve the temporal information of speech while reflecting its frequency domain characteristics. Therefore, researchers have attempted to use spectrograms instead of acoustic features for emotion classification. In the time domain, emotions are reflected across different time frames; in the frequency domain, different emotional information is distributed in the high and low frequency regions of speech. For example, anger exhibits rich acoustic features in high frequencies, while sadness exhibits rich acoustic features in low frequencies. Therefore, the importance of time frames and frequency intervals for emotional features differs in log-Mel spectrograms, thus requiring further research on how to extract significant time-frequency features. Summary of the Invention

[0004] In view of the above-mentioned defects in the existing technology, the present invention proposes a speech emotion recognition method based on time-frequency attention mechanism. The method has simple steps and can more accurately identify the emotion in speech by extracting time-frequency features related to emotion.

[0005] To achieve the above technical objectives, the present invention provides a speech emotion recognition method based on a time-frequency attention mechanism, comprising the following steps:

[0006] Step a: Extract log-Mel spectrogram features, use a cyclic padding method to pad the log-Mel spectrogram features to make them longer, and input the longer log-Mel spectrogram features into the speech emotion recognition model. The speech emotion recognition model includes a sequentially connected time-frequency convolutional module, a time-frequency attention module, a multi-layer convolutional network, and a fully connected layer.

[0007] Step b: The time-frequency convolution module captures the time-domain and frequency-domain variation information of the input log-Mel spectrogram, and extracts the time-frequency features of the log-Mel spectrogram at the same time;

[0008] Step c: Use the time-frequency attention module to learn the weights of the time-frequency features in both the time and frequency domains, label the sentiment features, and generate a time-frequency weighted feature map;

[0009] Step d: Feed the time-frequency weighted feature map into a multi-layer convolutional network to learn deep sentiment features;

[0010] Step e: Perform mean pooling along the time dimension and max pooling along the frequency dimension on the deep sentiment features to obtain one-dimensional sentiment features;

[0011] Step f: Feed the one-dimensional emotion features into a fully connected layer and project them to the desired dimension, and use a Softmax classifier to classify different speech emotions.

[0012] Furthermore, step a specifically includes:

[0013] Step a-1: The log-mel spectrogram is finally generated by performing pre-emphasis, framing, windowing, and short-time Fourier transform on the audio signal.

[0014] Step a-2: The time length of the input log-Mel spectrogram is wavtime. Set the input batch, mark the minimum time length of the log-Mel spectrogram in a batch as min, and calculate the maximum time length of the log-Mel spectrogram in the batch using the max() function and mark it as max.

[0015] Step a-3: If the maximum time length of the batch (max) is greater than or equal to the minimum time length (min) of the log-Mel spectrogram, then the maximum time length of the batch is max; otherwise, max equals min.

[0016] Step a-4: Determine if the time length wavtime of the input log-Mel spectrogram is equal to max. If true, return the log-Mel spectrogram feature with time length max.

[0017] Step a-5: If false, divide the maximum time length of the batch (max) by the input spectrum time length (wavtime) to obtain the length to be filled, and use the repeat() function to perform loop filling;

[0018] Step a-6: Return the padding feature to ensure that if the input log-Mel spectrogram is longer than wavtime, the maximum length is set, and the minimum length of each log-Mel spectrogram is wavtime.

[0019] Furthermore, the time-frequency convolution module is constructed by using two sets of filters of different shapes to learn time-frequency information: the first set is a time-domain convolutional filter TCNN, which obtains time variation information R along the time dimension of the log-Mel spectrogram. t The first group has a time dimension of T and a frequency dimension of 1; the second group is a frequency domain convolutional filter FCNN, which obtains frequency information R along the frequency dimension of the log-Mel spectrogram. f Its frequency dimension is F and its time dimension is 1. TCNN and FCNN together form the time-frequency convolution module TFCNN.

[0020] Furthermore, the specific steps for constructing the time-frequency attention module include:

[0021] Step c-1: Generate temporal attention weights and frequency domain attention weights: output temporal feature map F after passing through TCNN and FCNN. t Frequency domain feature map F f Each feature map has three dimensions: channel C, frequency H, and time dimension W. The output feature maps F are... t ∈R C×H×W and F f ∈R C×H×W The channel information of the feature map is aggregated using average pooling and max pooling methods to generate two two-dimensional maps: F avg ∈R 1×H×W F max ∈R 1×H×W Two two-dimensional mappings F avg ∈R 1×H×W F max ∈R 1×H×W The feature maps represent the average pooling and max pooling features of the entire channel, and then are processed by two independent convolutions with different kernel sizes. 1×t and conv f×1 The learning process involves using a convolutional kernel whose size represents the coverage area of ​​interactions within a given region. Finally, a sigmoid activation function is used to generate temporal attention weights T. w and frequency domain attention weight S w The formula is shown below:

[0022]

[0023]

[0024] In formula (1), F represents t Aggregated features after average pooling; F represents t Aggregated features after max pooling; conv 1×tThis represents a convolution with a kernel size of 1×t, where t represents the number of frames in the time region; T w Indicates the weights related to the time dimension; The activation function is Sigmoid; similarly, in formula (2), F represents f Aggregated features after average pooling; F represents f Aggregated features after max pooling; conv f×1 This represents a convolution with a kernel size of f×1, where f is the number of frequency intervals; S w Indicates the weights related to the frequency dimension;

[0025] Step c-2: Temporal feature map F t Frequency domain feature map F f A time-frequency attention module is constructed using residual connections. This module is a feedforward neural attention module, and attention mappings are derived in both the time and frequency domains. The attention is then mapped onto time-frequency features for adaptive feature refinement, namely the time-frequency attention mechanism TF_atten, which includes a time-domain attention mechanism T_atten and a frequency-domain attention mechanism F_atten. Finally, the time-frequency attention module is used to generate a time-frequency weighted feature map.

[0026] Using time-frequency attention weights (T) w S w ) and corresponding feature map (F t F f Element-wise multiplication is performed. To preserve the integrity of emotional information in the speech, residual connections are used in the calculation of the time-frequency attention-weighted feature maps (T, S). The calculation method for the time-frequency attention-weighted feature maps (T, S) is as follows:

[0027]

[0028]

[0029] in, This indicates element-wise multiplication.

[0030] Furthermore, the time-frequency weighted feature maps are concatenated using a concat method to obtain the fused feature F. ts It utilizes multi-layer convolution to learn deep sentiment features, where the multi-layer convolution consists of convolutional layers and pooling layers.

[0031] Beneficial effects:

[0032] This invention employs a time-frequency attention module to further improve speech emotion recognition performance. The time-frequency attention mechanism TF_atten extracts emotionally significant time-frequency features, while the frequency domain attention mechanism focuses on frequency intervals rich in emotional information. The time domain attention mechanism can locate multiple emotional regions, highlighting speech information areas. The two attention mechanisms work together to assist the emotion recognition model in extracting emotion-related time-frequency features from a variable-length log-Mel spectrogram. Simultaneously, residual connections facilitate information transmission, thereby more accurately recognizing emotions in speech. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the speech emotion recognition model based on the time-frequency attention mechanism of the present invention. Detailed Implementation

[0034] The embodiments of the present invention will now be described in detail with reference to specific examples and accompanying drawings.

[0035] like Figure 1 As shown, the specific steps of the speech emotion recognition method based on the time-frequency attention mechanism of the present invention are as follows:

[0036] Step 1: 2943 impromptu conversations (improved) from the IEMOCAP database were selected for the experiment. Excitement was merged into happiness, and four emotion categories—anger, sadness, happiness, and neutral—were used for emotion classification. The IEMOCAP database was divided into 10 subsets. Nine subsets were used alternately as training data, and one subset as test data. Ten-fold cross-validation was performed, and the classification accuracy was obtained for each trial. The final classification accuracy was obtained by averaging the results of the ten trials. This paper constructs a speaker-independent speech emotion recognition system. Unweighted precision (UA) and weighted precision (WA) were used.

[0037] Step 2: Perform frame-by-frame windowing on the speech signal, setting the frame length to 40 milliseconds and the frame shift to 10 milliseconds. For each frame of the speech signal, calculate the 2048-byte Discrete Fourier Transform (DFT) to obtain the short-time spectrum, and then square the modulus of the short-time spectrum to obtain the discrete power spectrum. Pass the discrete power spectrum through a Mel filter bank containing 128 Mel filters to calculate 128 Mel spectra. Then, calculate the natural logarithm of the Mel spectra to obtain the log-Mel spectrogram.

[0038] Step 3: Use the maximum length of each batch to cyclically fill the log-Mel spectrogram as model input.

[0039] Step 4: Log-Mel spectrogram features. First, time-frequency sentiment features F are extracted using the time-frequency convolutional module TFCNN. t ∈R C ×H×W and F f ∈R C×H×W Furthermore, the time-frequency attention module learns weights for sentiment features in both the time and frequency domains, calibrates the sentiment features, and extracts time-frequency weighted feature maps T and S. The structure of the time-frequency convolution module TFCNN is shown in Table 1.

[0040] Table 1. TFCNN network structure and specific parameters

[0041]

[0042]

[0043] Step 5: Fuse the time-frequency weighted feature maps T and S to obtain the fused feature F. ts The data is fed into a multi-layer CNN to learn deep sentiment features. The structure and parameters of the multi-layer CNN are shown in Table 2.

[0044] Table 2 CNN layer network structure parameters

[0045]

[0046] Step 6: Perform mean pooling and max pooling operations on the deep sentiment features obtained in Step 5 along the time dimension and frequency dimension respectively to obtain a one-dimensional sentiment feature of size 512.

[0047] Step 7: Feed the one-dimensional features from Step 6 into a fully connected layer for dimensionality reduction, and then perform sentiment classification using Softmax.

[0048] To further illustrate the present invention, a comparative experiment was designed. The specific experiment is as follows:

[0049] ①CNN: This experiment uses a cyclically filled log-Mel spectrogram as input and uses a traditional CNN network to extract local sentiment features as a comparative experiment. The CNN structure and parameters are shown in Table 2.

[0050] ②TFCNN: The TCNN and FCNN convolution kernel sizes used in this experiment are 1×11 and 11×1, respectively. The TFCNN network structure and specific parameters are shown in Table 1. The time-frequency features are concatenated by Concat, and then deep sentiment features are learned by multi-layer CNN. The CNN network structure and specific parameters are shown in Table 2. Finally, sentiment classification is performed through fully connected layers.

[0051] ③TF_atten: This experiment uses T_atten and F_atten to extract time-frequency features based on (2). The kernel sizes of T_atten and F_atten are 1*21 and 21*1, respectively, for subsequent experiments;

[0052] The experimental results are shown in Table 3.

[0053] Table 3 Experimental results of different models

[0054]

[0055] Comparing TFCNN and CNN, the experimental results show that TFCNN achieves a WA of 75.8%, a 2.6% improvement over CNN, and an UA of 74.8%, a 3.9% improvement over CNN. These experiments demonstrate that the time-frequency features extracted using TFCNN have better sentiment classification performance than the local features extracted using traditional CNNs. Comparing the experimental results of TF_atten and TFCNN, TF_atten achieves WA and UA improvements of 1.2% and 1.1% respectively compared to TFCNN. This is attributed to the collaboration between time-frequency attention mechanisms. The frequency domain attention mechanism focuses on frequency intervals rich in emotional information, while the time domain attention mechanism locates multiple emotional regions, highlighting speech information regions and focusing on speech frames rich in emotional information. The combined effect of these two attention mechanisms helps the sentiment recognition model effectively extract emotion-related time-frequency features from variable-length log-Mel spectrograms, validating the effectiveness of the emotion-significant time-frequency features extracted using the time-frequency attention mechanism.

[0056] The above description of the embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention. It should be noted that those skilled in the art can make various modifications and alterations to the present invention without departing from the principle of the present invention, and these improvements also fall within the protection scope of the claims of the present invention.

Claims

1. A method for speech emotion recognition based on time-frequency attention mechanism, characterized in that, The steps are as follows: Step a: Extract log-Mel spectrogram features, use a cyclic padding method to pad the log-Mel spectrogram features to make them longer, and input the longer log-Mel spectrogram features into the speech emotion recognition model. The speech emotion recognition model includes a sequentially connected time-frequency convolutional module, a time-frequency attention module, a multi-layer convolutional network, and a fully connected layer. Step b: The time-frequency convolution module captures the time-domain and frequency-domain variation information of the input log-Mel spectrogram, and extracts the time-frequency features of the log-Mel spectrogram at the same time; Step c: Use the time-frequency attention module to learn the weights of the time-frequency features in both the time and frequency domains, label the sentiment features, and generate a time-frequency weighted feature map; Step d: Feed the time-frequency weighted feature map into a multi-layer convolutional network to learn deep sentiment features; Step e: Perform mean pooling along the time dimension and max pooling along the frequency dimension on the deep sentiment features to obtain one-dimensional sentiment features; Step f: Input the one-dimensional emotion features into a fully connected layer and project them to the desired dimension, and use a Softmax classifier to classify different speech emotions; The construction of the time-frequency attention module includes: Step c-1: Generate temporal attention weights and frequency domain attention weights: output temporal feature maps after passing through TCNN and FCNN. and frequency domain feature map Each feature map has three dimensions: channel C, frequency H, and time dimension W. The output feature maps are two... and The channel information of the feature map is aggregated using average pooling and max pooling methods to generate two two-dimensional maps: , Two two-dimensional mappings , The feature maps represent the average pooling and max pooling features of the entire channel, achieved through independent convolutions with two different kernel sizes. and The learning process is performed, and finally, temporal attention weights are generated using the Sigmoid activation function. Frequency domain attention weights The formula is shown below: （1）， （2）， In formula (1), express Aggregated features after average pooling; express Aggregated features after max pooling; Indicates the kernel size as The convolution of , where t represents the number of frames in the time region; Indicates the weights related to the time dimension; The activation function is Sigmoid; similarly, in formula (2), express Aggregated features after average pooling; express Aggregated features after max pooling; Indicates the kernel size as The convolution of f, where f is the number of frequency intervals; Indicates the weights related to the frequency dimension; Step c-2: Temporal feature map and frequency domain feature map A time-frequency attention module is constructed using residual connections. This module is a feedforward neural attention module, and attention mappings are derived in both the time and frequency domains. The attention is then mapped onto time-frequency features for adaptive feature refinement, namely the time-frequency attention mechanism TF_atten, which includes a time-domain attention mechanism T_atten and a frequency-domain attention mechanism F_atten. Finally, the time-frequency attention module is used to generate a time-frequency weighted feature map. Using time-frequency attention weights ( , ) and corresponding feature maps ( , Element-wise multiplication is performed. To preserve the integrity of emotional information in the speech, residual connections are used in the calculation of the time-frequency attention-weighted feature maps (T, S). The calculation method for the time-frequency attention-weighted feature maps (T, S) is as follows: （3）， （4）， wherein represents the multiplication of elements.

2. The speech emotion recognition method based on time-frequency attention mechanism according to claim 1, characterized in that Step a specifically includes: Step a-1: The log-mel spectrogram is finally generated by performing pre-emphasis, framing, windowing, and short-time Fourier transform on the audio signal. Step a-2: The time length of the input log-Mel spectrogram is wavtime. Set the input batch, mark the minimum time length of the log-Mel spectrogram in a batch as min, and calculate the maximum time length of the log-Mel spectrogram in the batch using the max() function and mark it as max. Step a-3: If the maximum time length of the log-Mel spectrogram in this batch is greater than or equal to the minimum time length of the log-Mel spectrogram, then the maximum time length of the log-Mel spectrogram in this batch is max; otherwise, max equals min. Step a-4: Determine if the time length wavtime of the input log-Mel spectrogram is equal to max. If true, return the log-Mel spectrogram feature with time length max. Step a-5: If false, divide the maximum time length of the batch (max) by the input spectrum time length (wavtime) to obtain the length to be filled, and use the repeat() function to perform loop filling; Step a-6: Return the padding feature to ensure that if the input log-Mel spectrogram is longer than wavtime, the maximum length is set, and the minimum length of each log-Mel spectrogram is wavtime.

3. The speech emotion recognition method based on time-frequency attention mechanism according to claim 1, characterized in that The time-frequency convolution module is constructed by using two sets of filters of different shapes to learn time-frequency information: the first set is a time-domain convolutional filter (TCNN), which obtains time-varying information along the time dimension of the log-Mel spectrogram. The first group has a time dimension of T and a frequency dimension of 1; the second group is a frequency domain convolutional filter FCNN, which obtains frequency information along the frequency dimension of the log-Mel spectrogram. Its frequency dimension is F and its time dimension is 1. TCNN and FCNN together form the time-frequency convolution module TFCNN.

4. The speech emotion recognition method based on time-frequency attention mechanism according to claim 1, characterized in that The time-frequency weighted feature maps are spliced by Concat to obtain fused features Deep emotional features are learned by using multi-layer convolution, where the multi-layer convolution is composed of a convolution layer and a pooling layer.

Images