Speech emotion recognition method based on multi-scale attention

Abstract

The invention relates to the technical field of speech emotion recognition, and particularly discloses a speech emotion recognition method based on a multi-scale attention mechanism. The method comprises the following steps: preprocessing a voice sample, performing frequency disturbance enhancement on voice by adopting a VTLP algorithm, performing segment-level oversampling on a voice segment, and extracting Logfbank features; inputting the processed speech features into a shallow convolutional network to extract basic emotion features; a multi-scale convolution structure is adopted to carry out multi-granularity modeling on shallow layer features, emotional features of different scales are extracted, and spatial attention and channel attention mechanisms are fused to focus on a key area; further sending the fusion features into a deep convolutional network to extract high-level semantic information; and finally realizing sentiment classification through global average pooling and a full connection layer. According to the method, the perception capability of the model for fine-grained emotion clues is enhanced through the multi-scale receptive field, the robustness and recognition precision of the model in a data imbalance scene are improved, and the method is suitable for construction of a high-performance speech emotion recognition system.

Description

Technical Field

[0001] The present invention relates to the fields of speech emotion recognition and deep learning, and in particular to a speech emotion recognition method based on a multi-scale attention mechanism. Background Art

[0002] With the continuous development of applications such as human-computer interaction, intelligent customer service, and emotional computing, speech emotion recognition (SER) has become an important research direction in natural language processing and speech processing. Existing speech emotion recognition methods mainly rely on deep neural networks to model and classify emotional patterns in speech signals. Common model structures include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and attention mechanism modules.

[0003] For example, patent publication CN108717856B proposes a speech emotion recognition method based on a combination of multi-scale convolution and recurrent neural networks. This method captures multi-scale emotional features through different receptive fields, improving the model's ability to model both local and long-term dependent emotional information. In academic research, some literature has also attempted to introduce spectral perturbations (such as VTLP) as a data augmentation strategy to improve model robustness. Some studies have introduced regional attention or self-attention mechanisms to enhance the model's ability to focus on key speech regions.

[0004] Although existing methods have made some progress in emotion recognition accuracy, the following technical limitations still exist:

[0005] 1. Most methods use convolution kernels with fixed receptive fields, which cannot effectively integrate emotional features of different scales and have difficulty capturing both fine-grained and global emotional cues.

[0006] 2. The attention mechanism usually only works on single-scale feature maps and fails to effectively integrate with multi-scale structures, limiting the model's perception capabilities.

[0007] 3. Current data augmentation strategies generally lack the ability to address data imbalance. During training, insufficient samples of minority emotion categories can easily cause the model to favor the dominant emotion category.

[0008] 4. Traditional deep models have difficulty effectively distinguishing segment-level speech fragments with relatively subtle emotional changes, affecting the stability and practicality of the recognition system.

[0009] Therefore, there is an urgent need for a speech emotion recognition method that integrates multi-scale feature extraction mechanism and multi-dimensional attention mechanism, and combines multi-dimensional data enhancement and category balancing strategy to enhance the model's emotional expression ability and sample generalization ability, thereby improving robustness and accuracy in practical applications. Summary of the Invention

[0010] In view of the shortcomings of the traditional single-scale model, such as the difficulty of fully expressing emotional features in a single receptive field, the present invention proposes and designs a speech emotion recognition method based on multi-scale attention to solve the above problems. The invention mainly includes two parts: 1. Data preprocessing method: For the speech sample data, data enhancement is performed through the VTLP algorithm, and the segment-level speech sample oversampling method is used to balance the number of segment-level speech categories. 2. Multi-scale attention module: A module is designed to extract emotional features of different granularities. The module uses convolutional layers of different scales to extract speech emotional features of different granularities, and then combines spatial attention and channel attention to focus on the key features in speech emotion.

[0011] The specific steps are as follows:

[0012] (1) Speech data preprocessing. First, the speech data samples are enhanced using the VTLP algorithm. Then, the speech samples are divided into speech segments of equal length and normalized using the local normalization method. Finally, segment-level sample oversampling is performed to balance the emotion categories of the speech segments. The Logfbank feature extraction algorithm is used to obtain the corresponding speech sample features as the input of the model.

[0013] (2) Perform shallow feature mapping on speech data. Input the preprocessed speech data and obtain shallow speech emotion features through two stacked convolutional layers.

[0014] (3) Multi-scale attention fusion. By inputting feature maps into convolution kernels of different sizes, feature maps of different granularities are obtained, and these feature maps are fused at multiple scales. Combining spatial attention and channel attention mechanisms, the model is able to perceive important emotional areas, thereby enhancing the ability to extract key information from sample data.

[0015] (4) Deep feature extraction. Imitating the human multi-level deepening emotion processing mechanism, the feature map obtained in step (3) is input into the deep feature extraction module formed by stacking multiple convolutional layers to extract the deep emotional features in the speech sample.

[0016] (5) Classification by classifier: The input is sent to a classifier consisting of a global average pooling layer and a fully connected layer to classify the speech emotion category.

[0017] Compared with existing speech emotion recognition methods, the advantages of the present invention are:

[0018] (1) This paper designs a speech emotion recognition model based on a multi-scale attention mechanism. Through the multi-scale fusion mechanism, convolution kernels of different sizes are used to extract speech emotion from speech samples to achieve the effect of multi-scale feature complementarity. The spatial attention and channel attention mechanisms enable the model to have the ability to perceive important emotional areas, thereby ignoring unimportant redundant information in the speech.

[0019] (2) A preprocessing method for speech data was designed. In the preprocessing step, the VTLP algorithm was used to perform data enhancement on each speech training sample. The method of undersampling a small number of speech category segments was also used to balance the number of samples of each emotion category, avoiding the impact of the model learning effect due to the imbalance of emotion categories in the speech segment samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 :Overall architecture diagram of speech emotion recognition model based on multi-scale attention

[0021] Figure 2 :Shallow feature mapping module structure diagram

[0022] Figure 3 : Multi-scale attention module structure diagram

[0023] Figure 4 :Deep feature extraction module structure diagram DETAILED DESCRIPTION

[0024] This paper designs a speech emotion recognition model based on multi-scale attention. The overall architecture of the model is as follows: Figure 1 As shown, the specific implementation is as follows:

[0025] 1. Data Preprocessing

[0026] First, the speech data is preprocessed, using the Vocal Tract Length Perturbation (VTLP) algorithm to enhance the training speech and increase the training data sample. The VTLP algorithm randomly generates a distortion factor for each sentence, distorting the frequency axis of the spectrum to remove the impact of vocal tract differences on speech recognition results. Furthermore, the VTLP algorithm adds Gaussian white noise to the original speech and modulates the audio volume and pitch to generate more diverse speech samples.

[0027] All data-augmented speech data was then segmented into 3-second segments and normalized based on the segment data. The specific processing method was as follows: for each speech segment, the absolute value was sorted. The 99.9th percentile position in ascending order was used as the reference value, and each value in the speech segment was divided by this reference value. Finally, the data in the speech segment was truncated to a value in the range [-1, 1]. For speech segments less than 3 seconds in length, the beginning and end were symmetrically padded with zero values. All speech segments were subjected to the Logfbank feature extraction algorithm to obtain the corresponding feature maps.

[0028] Finally, random oversampling is performed on the truncated speech segment dataset. By randomly selecting some samples from the minority class, copying the selected samples and adding them to the original dataset, the original dataset is expanded and the data is balanced.

[0029] 2. Shallow Feature Mapping

[0030] Its overall structure is as follows Figure 2 As shown in Figure 2, it consists of two stacked convolutional modules. Each convolutional module consists of a 3x3 convolutional layer, a batch normalization layer, and a ReLU activation function. The shallow feature mapping module extracts low-dimensional emotional features from the original speech feature map through two convolutional layers. To increase the number of low-level features learned by the model, the number of channels in the feature map increases as it passes through this module.

[0031] To allow the convolutional layer to map features to a high-dimensional space and help subsequent modules extract further features, a batch normalization layer is added after each convolutional layer. The data distribution statistics are calculated based on each batch of input data, and normalization is performed based on these statistics to enhance the network's tolerance to input changes. The calculation formula for batch normalization is shown in (1).

[0032]

[0033] Among them, x out represents the output after batch normalization, γ and β represent the learnable network parameters, μ represents the mean of the speech features in each batch, σ represents the standard deviation of the speech features in each batch, and ε is a very small positive number that avoids the extreme case where the denominator is zero.

[0034] After batch normalization, the ReLU activation function is used to activate the network, which maintains linearity when the input is positive and outputs zero when the input is negative. This activation function is not only computationally efficient but also alleviates the vanishing gradient problem.

[0035] 3. Multi-scale Attention Fusion

[0036] Multi-scale attention fusion consists of a multi-scale fusion module, a spatial attention module, a channel attention module, a residual connection part, and a downsampling module. The modules are described as follows:

[0037] 3.1 Multi-Scale Fusion Module. This module consists of five convolutional layers of different sizes, each followed by a batch normalization layer and a Reluctant Unit (ReLU) activation function. Feature maps are fed into five convolutional layers of sizes 1*1, 3*3, 5*5, 7*7, and 9*9, respectively, to generate feature maps with different receptive fields. Finally, the feature maps from different receptive fields are element-wise summed to fuse sentiment features of different granularities.

[0038] 3.2 Spatial Attention Module: This module consists of a global maximum pooling layer, a 7*7 convolutional layer, and a 1*1 convolutional layer.

[0039] In the global maximum pooling layer, for each position in the feature map, the maximum value of all corresponding channels is taken, which plays the role of assigning weights to each position in the feature map and obtaining the importance of each area. Its calculation formula is shown in (2).

[0040]

[0041] Where N represents the number of channels, Indicates taking the maximum value of all channels corresponding to the (x, y) position in the feature map, (x ′ ,y ′ ) indicates the corresponding output result.

[0042] After global max pooling, features are further extracted through a 7*7 convolutional layer. The Sigmoid activation function is used for activation to introduce nonlinear factors. The calculation formula is shown in (3).

[0043]

[0044] The feature map obtained in the above steps is multiplied element-by-element with the feature map output by the multi-scale fusion module to achieve the effect of incorporating the spatial attention mechanism into the feature map after multi-scale fusion.

[0045] 3.3 Channel Attention Module. It consists of a global average pooling layer, a 1*1 convolutional layer, and a ReLU activation function. The global average pooling layer takes the average of each channel in the feature map to obtain the importance of each channel. Its calculation formula is shown in (4).

[0046]

[0047] Among them, H, W represent the height and width of the feature map respectively, c represents the channel index, y c Represents the output corresponding to channel c.

[0048] After global average pooling, the network passes through a 1x1 convolutional layer and is activated using the ReLU activation function. When the input is negative, the output value is 0, suppressing unimportant channels; when the input is positive, the output value remains unchanged, enhancing channel attention. The calculation formula is shown in (5).

[0049] f(x)=max(0,x) (5)

[0050] 3.4 Residual Connection. The output of the spatial attention module and the output of the channel attention module are multiplied element-by-element to perform attention feature fusion. The attention feature fusion result is then added element-by-element to the output feature map of the shallow feature mapping module in step 2 to achieve the effect of introducing multi-scale attention on the feature map.

[0051] 3.5 Downsampling Module. The downsampling module adjusts the number of feature channels and reduces the size of the feature map. This module consists of a 1x1 convolutional layer, a 2x2 max pooling layer, a ReLU activation function, and a batch normalization layer. The 1x1 convolutional layer adjusts the feature map dimensions. The max pooling layer compresses the data and reduces computational complexity.

[0052] 4. Deep feature extraction module.

[0053] The multi-scale attention-weighted feature map is input into the deep feature extraction module, which further abstracts the deeper features of the speech. The deep feature extraction module consists of four stacked convolutional modules. Each convolutional module consists of a 3x3 convolutional layer, a batch normalization layer, a ReLU activation function, and an average pooling layer with a kernel size of 2x2 and a stride of 2. Inspired by the step-by-step deepening of human emotion processing, the number of channels in the output feature map increases with each convolutional module. Specifically, as the feature map passes through the deep feature extraction module, its number of channels increases from an initial 48 to 64, 96, and 128, respectively, with the final output having 160 channels.

[0054] In practical applications, the deep feature extraction module structure can flexibly adjust the number of convolutional layer stacking layers and the parameter size of the convolution kernel according to the latency and accuracy requirements of specific tasks, and has good scalability.

[0055] 5. Sentiment classifier.

[0056] The feature map obtained in the previous step is input to a classifier consisting of a global average pooling layer and a fully connected layer for speech emotion classification. Global average pooling compresses the feature information of each channel. The fully connected layer outputs a number of dimensions that matches the number of emotions to be classified, converting the feature information into corresponding emotion categories. Model training uses the cross-entropy loss function to measure the difference between the classifier's classification and the true classification.

Claims

1. A speech emotion recognition method based on a multi-scale attention mechanism, characterized in that: The following steps are involved: (1) Perform voice frequency perturbation data enhancement (VTLP) on the speech samples; (2) Divide the enhanced sample into speech segments of equal length and normalize the features; (3) Oversampling technology is used for speech segments with a few emotional categories to balance the category distribution; (4) Extract Logfbank features as model input; (5) Use two layers of convolution to extract shallow features from input features; (6) Input shallow features into the multi-scale attention fusion module, which includes: (a) Parallel multi-scale convolution operations are performed to obtain multi-scale feature maps. Feature maps in different scale receptive fields are added element by element to obtain the scale-fused feature map. (b) Use the spatial attention weight generation module to spatially weight the fused features; (c) Use the channel attention mechanism to perform channel weighting on the fused feature map; (d) Multiply the output of the spatial attention module and the output of the channel attention module element by element to perform attention feature fusion; (e) The attention feature fusion result is added element by element to the shallow feature map in (5), and then the number of feature channels and feature size are adjusted through the downsampling module to obtain the weighted feature; (7) Input the weighted features into the deep convolution module to continue extracting deep features; (8) Use global average pooling and fully connected layers to output sentiment classification results.

2. The method according to claim 1, wherein The sizes of the multi-scale convolution kernel include 1×1, 3×3, 5×5, 7×7 and 9×9, which are used to capture emotional feature information of different granularities in speech.

3. The method according to claim 1, wherein The spatial attention mechanism generates a spatial weight map by performing maximum pooling on the input feature map, passing through a 7×7 convolution layer and a Sigmoid activation function.

4. The method according to claim 1, wherein The channel attention mechanism uses a global average pooling layer, a 1×1 convolutional layer, and a ReLU activation function to generate a channel weight map.

5. The method according to claim 1, wherein The method of randomly oversampling minority class samples of the segment-level speech samples is used to alleviate the problem of emotional class imbalance.

6. The method according to claim 1, wherein The deep feature extraction module includes no less than 4 convolutional layers, and each layer is connected to a batch normalization layer and a nonlinear activation function.

7. The method according to claim 1, wherein The number of classifier output nodes is equal to the total number of emotion categories, which include neutral, happy, sad, angry, etc.

8. The method according to claim 1, wherein The frequency perturbation coefficient α of the VTLP algorithm ranges from 0.9 to 1.1, and is used to simulate the natural variation of the speaker's vocal system.

9. The method according to claim 1, wherein The method is applicable to speech emotion recognition scenarios in Chinese, English or other languages, and has cross-language transfer capabilities.

Images