A method for emotion recognition in noisy speech based on deep learning

Abstract

The present invention belongs to the field of speech signal processing in deep learning, and specifically relates to a noisy speech emotion recognition method based on deep learning. The method firstly integrates multiple learnable features as the input of the model to improve the learning performance related to the model task; secondly, the operation of mixed spectrogram blocking is introduced, and multi-head attention is calculated block by block to capture local emotion information. Experiments have shown that the blocking operation can effectively avoid the interference problem caused by local noise; finally, the present invention integrates the global context information of the attention calculated frame by frame with the local emotion information, which promotes the model's understanding of the context semantics and improves the classification accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of speech signal processing in deep learning, and specifically relates to a noisy speech emotion recognition method based on deep learning. Background Art

[0002] Speech emotion recognition is a computer simulation of the process of human emotion perception and understanding. Its task is to extract acoustic features that express emotions from the collected speech signals, find the mapping relationship between these acoustic features and human emotions, and then recognize the speech signals through the mapping relationship to achieve the purpose of human-computer interaction. Speech emotion recognition usually includes two parts: feature extraction and classification. Specifically, the speech signal will be converted into various readable physical features by the speech processing system, and feature vectors that are beneficial to modeling will be obtained through feature extraction; the classification model constructed by the classifier is used for emotion classification through feature vector training. The classifier includes both traditional machine learning SVM, decision trees, etc., as well as deep learning models such as DNN, CNN, and LSTM.

[0003] In existing speech emotion recognition technology solutions, static Mel-spectrogram, Mel-frequency cepstral coefficient (MFCC) and other fixed features extracted from the original speech are generally used, and then deep learning technologies such as CNN and LSTM are used to model the emotional features, and finally the speech emotions are classified. However, the calculation formulas in these feature extraction methods are fixed, the extracted features are relatively redundant, and it is impossible to learn information that is beneficial to the task from the original speech according to the characteristics of the task. In addition, the classification models in existing speech emotion recognition all model the speech as continuous time series information, which makes it difficult to have good recognition accuracy for actual business scenario data containing noise. Summary of the invention

[0004] To solve the above problems, the present invention provides a method for noisy speech emotion recognition based on deep learning, which specifically includes the following steps:

[0005] S1. Acquire audio data and preprocess it to obtain preprocessed data;

[0006] S2. Extract the Mel-spectrogram features and time-frequency features of the preprocessed data, and randomly mask the frequency domain and time domain of the Mel-spectrogram features to obtain random masking features;

[0007] S3. Construct a speech emotion recognition network and train the speech emotion recognition network using a hybrid feature obtained by combining time-frequency features and random masking features;

[0008] S4. Use the trained speech emotion recognition network to recognize the speech information to be tested.

[0009] Furthermore, the process of obtaining mixed features is:

[0010] Extract the Mel-spectrogram features of the preprocessed data, expressed as:

[0011] M = log(melW·(STFT(X)) 2 )

[0012] Randomly masking the frequency domain and time domain of the Mel-spectrogram feature obtains the random masking feature, which is expressed as:

[0013] M′=Specaugment(M)

[0014] The time-frequency features of the preprocessed data are extracted using stacked residual one-dimensional convolution, expressed as:

[0015] N = StackedConv1D(X)

[0016] The time-frequency feature is combined with the random masking feature to obtain the hybrid feature, which is expressed as:

[0017] F = concat([M′,N], dim = 1)

[0018] Where X∈R (B,T) Represents preprocessed data, STFT() represents fast Fourier transform, melW represents Mel filter bank, Specaugment() represents random masking of time domain and frequency domain data enhancement operations, StackedConv1D() represents residual one-dimensional convolution, concat() represents concatenation, M represents Mel-spectrogram features, M′ represents random masking features, N represents time-frequency features, and F represents mixed features.

[0019] Furthermore, a backbone model is provided in the speech emotion recognition network, which is composed of multiple repeated modules with the same structure, and the input of each repeated module is dynamically changed. Specifically, the stacked module includes a layer normalization unit, a multi-head block attention calculation unit, a frequency domain compression and channel enhancement unit, a downsampling and channel expansion module, and a feedforward neural network unit.

[0020] Furthermore, the processing process of the frequency domain compression and channel enhancement unit is as follows:

[0021] Compress the output of the multi-head attention unit in the frequency domain to obtain compressed features;

[0022] The compressed features are self-attentioned frame by frame, and the output of the frame-by-frame self-attention calculation is channel-enhanced with a smoothing factor.

[0023] Furthermore, the training process of the speech emotion recognition network includes:

[0024] S11. Use two-dimensional convolution to embed the mixed features into time and frequency to obtain high-dimensional features;

[0025] S12. Absolute position encoding is performed on each audio frame in the high-dimensional feature;

[0026] S13. Divide the high-dimensional features after absolute position encoding into blocks to obtain multiple spectrum blocks;

[0027] S14. Input all spectrum blocks into the backbone model for calculation until all spectrum blocks are calculated;

[0028] S15. After performing global average pooling on the calculation results of S14, the cross entropy with a smoothing factor is used to calculate the loss and back propagate to optimize the parameters of the speech emotion recognition network.

[0029] Furthermore, the cross entropy with smoothing factor is expressed as:

[0030]

[0031]

[0032]

[0033] Among them, q i represents the predicted probability of the i-th category, K represents the total number of sentiment categories, α represents the smoothing factor, target represents the original label, and z i Represents the result after global average pooling.

[0034] Beneficial effects of the present invention:

[0035] The present invention provides a noisy speech emotion recognition method based on deep learning, which uses learnable mixed features as modeling input, which can learn not only the features that can be perceived by human hearing, but also the richer and more diverse time-frequency features. In addition, the learnable mixed features can also learn information that is useful for the task from the original speech according to the characteristics of the task, abandoning hundreds of fixed features and providing more valuable information for the network.

[0036] The present invention adopts a unique spectrum graph segmentation technology to calculate local self-attention for each spectrum block, which can effectively weaken the interference caused by local noise while capturing real local emotional information. In addition, the modeling of the present invention also considers the impact of global contextual information on emotional classification, and uses a frame-by-frame self-attention mechanism to capture global dependencies and improve classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the noisy speech emotion recognition method of the present invention;

[0038] Figure 2 A diagram of a stacked one-dimensional convolution structure of the present invention;

[0039] Figure 3 This is a network structure diagram of noisy speech emotion recognition of the present invention;

[0040] Figure 4 This is a flow chart of the network training for noisy speech emotion recognition of the present invention;

[0041] Figure 5 It is a block schematic diagram of the present invention;

[0042] Figure 6 It is a processing diagram of the frequency domain compression and channel enhancement unit of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In order to extract effective speech information in a complex speech environment, the present invention provides a speech emotion recognition method based on deep learning, which models speech information through deep learning to automatically identify the speaker's emotion category, such as Figure 1 As shown, the following steps are included:

[0045] S1. Acquire audio data and preprocess it to obtain preprocessed data;

[0046] S2. Extract the Mel-spectrogram features and time-frequency features of the preprocessed data, and randomly mask the frequency domain and time domain of the Mel-spectrogram features to obtain random masking features;

[0047] S3. Construct a speech emotion recognition network and train the speech emotion recognition network using a hybrid feature obtained by combining time-frequency features and random masking features;

[0048] S4. Use the trained speech emotion recognition network to recognize the speech information to be tested.

[0049] Specifically, the present invention uses learnable mixed features as modeling input, which can learn features that can be perceived by human hearing, as well as rich and diverse time-frequency features. In addition, the learnable mixed features can also learn information that is useful for the task from the original speech according to the characteristics of the task, discarding redundant fixed features, and ensuring that more valuable information is provided to the subsequent model. As for this feature extraction part, SincNet, Dilated CasualConvolution, etc. can also be used as feature extraction modules to extract learnable features as model input.

[0050] In one embodiment, audio data is obtained from an offline database such as a cloud or a call center, where the amount of the audio data is related to an actual business scenario. The audio data is processed, that is, batches of offline data are converted to the same sampling frequency, and corresponding emotion tags are labeled according to actual business needs of the scenario, thereby obtaining preprocessed data; Mel-spectrogram features are extracted from the preprocessed data, and the frequency domain and time domain of the features are randomly masked for data enhancement. At the same time, stacked residual one-dimensional convolution (StackedConv1D) is used to directly extract time-frequency features from the original waveform, that is, the preprocessed data. Finally, the time-frequency features and Mel-spectrogram features of the preprocessed data are combined to obtain mixed features.

[0051] Specifically, the two components used to extract time-frequency features and Mel-spectrogram features will be trained together with the subsequent speech emotion recognition network. The calculation process for obtaining the mixed features is:

[0052] Extract the Mel-spectrogram features of the preprocessed data, expressed as:

[0053] M = log(melW·(STFT(X)) 2 )

[0054] Randomly masking the frequency domain and time domain of the Mel-spectrogram feature obtains the random masking feature, which is expressed as:

[0055] M′=Specaugment(M)

[0056] The time-frequency features of the preprocessed data are extracted using stacked residual one-dimensional convolution, expressed as:

[0057] N = StackedConv1D(X)

[0058] The time-frequency feature is combined with the random masking feature to obtain the hybrid feature, which is expressed as:

[0059] F = concat([M′,N], dim = 1)

[0060] Where X∈R (B,T) Represents preprocessed data, STFT() represents fast Fourier transform, melW represents Mel filter bank, Specaugment() represents random masking of time domain and frequency domain data enhancement operation, StackedConv1D() represents residual one-dimensional convolution, concat() represents concatenation combination, log represents logarithmic calculation, M represents Mel-spectrogram feature, M′ represents random masking feature, N represents time-frequency feature, and F represents mixed feature.

[0061] Specifically, the stacked residual one-dimensional convolution (StackedConv1D) structure is as follows Figure 2 As shown in Figure 1, three residual one-dimensional convolution modules with the same structure are stacked, and each residual one-dimensional convolution module includes four one-dimensional convolution layers and two activation functions.

[0062] In one embodiment, a speech emotion recognition network is constructed, and the speech emotion recognition network is trained using mixed features, wherein the structure of the speech emotion recognition network is as follows: Figure 3 As shown in the figure, it includes a time-frequency embedding module (Time-FrequencyEmbedding), an absolute position encoding module (Positional Encoding), a hybrid feature map block module (Block) and a backbone model. The backbone model includes multiple repeated modules with the same structure. The repeated modules include a layer normalization unit (Norm), a multi-head block attention calculation unit (Multi-Head BlockAttention), a frequency domain compression and channel enhancement unit (FCCE), a downsampling and channel expansion module (DSCE) and a feedforward neural network unit (Feed Forward). In the backbone model, the downsampling and channel expansion module is called once every time the input data passes through two repeated modules, so that the size and quantity of the output data are different from the input data, so the input of the repeated module is dynamically changed.

[0063] Specifically, Figure 4 As shown, the training process of the speech emotion recognition network includes: using two-dimensional convolution (Conv2D) to embed the mixed features in a higher dimension in the time-frequency embedding module to obtain high-dimensional features including three dimensions: channel (C), time domain (T), and frequency domain (D); considering the unique time series information contained in the audio data, the present invention adds additional position coding to each audio frame so that the network is sensitive to the position information of each frame, that is, absolute position coding is performed on each audio frame in the high-dimensional features, and the absolute position coding is calculated as follows:

[0064] APE(p,2i)=sin(p / 10000 2i / D )

[0065] APE(p,2i+1)=cos(p / 10000 2i / D )

[0066] Among them, p represents position, i represents dimension, and APE represents the absolute position encoding information added to each audio frame.

[0067] Specifically, the high-dimensional features after absolute position encoding are divided into blocks in the hybrid feature map block module to obtain multiple spectrum blocks, such as Figure 5 As shown, each spectrum block is represented as where num blocks The number of spectrum blocks obtained by blocking a high-dimensional feature. h and w represent the length and width of the block respectively. Different high-dimensional features may have different block sizes.

[0068] Specifically, all spectrum blocks are input into the backbone model for parallel calculation, and each spectrum block is calculated separately. The calculation process of each spectrum block is:

[0069] S11. The spectrum block is layer-normalized by a layer normalization unit, and then the multi-head attention of the spectrum block is calculated by a multi-head block attention calculation unit. This calculation method enables the speech emotion recognition network to use the characteristics of self-attention weighted average to weaken the noise while capturing local emotion information;

[0070] S12. The frequency domain compression and channel enhancement unit performs frame-by-frame attention calculation on the output of the multi-head block attention calculation unit. This module promotes the capture of global context information by the speech emotion recognition network and improves the emotion discrimination accuracy of the speech emotion recognition network.

[0071] S13. In order to ensure stable training and fast convergence of the speech emotion recognition network, a feedforward neural network unit is connected after the frequency domain compression and channel enhancement unit, and a residual connection method is adopted;

[0072] S14. Because emotion recognition requires more advanced abstract features than sequence tasks, the present invention calls the downsampling and channel expansion module once after every two repeated modules to perform feature map downsampling and expand the number of channels by one time.

[0073] Specifically, the multi-head attention calculation of the spectrum block is expressed as:

[0074] M ln =LN(M)

[0075]

[0076] Among them, LN represents layer normalization, For M lnThe query, key, and value parameter matrices obtained by linear mapping. In order to promote the attention map, that is, to calculate the further deviation of the intermediate variables in the block attention process, so that the model has more discrimination for each category, the present invention introduces the relative position deviation RPB of each pixel in each block. When calculating RPB, the relative position information is obtained by querying the initialized two-dimensional index matrix. The multi-head attention calculation output of the spectrum block is expressed as M * ∈R (B,T,D,C) .

[0077] Specifically, Figure 6 As shown, the processing process of the frequency domain compression and channel enhancement unit is:

[0078] The output of the multi-head attention unit is compressed in the frequency domain to obtain the compressed features and defined as

[0079]

[0080] Among them, x i,j,k Indicates M * A piece of data in; i∈[0,T),j∈[0,D),k∈[0,C); x′ ik ∈M * ′,M * ′ represents the feature map after compression in the frequency domain; next, the compressed feature M * ' performs frame-by-frame self-attention calculation, expressed as:

[0081]

[0082]

[0083] In order to provide adaptive gain to the original feature map, the output T of the frame-by-frame self-attention calculation is * Perform channel enhancement with a smoothing factor, expressed as:

[0084] Y=σ(T * W T +b)

[0085] Z=Y·αT *

[0086] Among them, σ represents the ReLU activation function; α∈(0,1]; specifically, Figure 3 As shown, after the frequency domain compression and channel enhancement unit, a feedforward neural network (FeedForward) is used for feedforward operation, and the calculation formula of the residual connection (Add) is:

[0087]

[0088] Z r =Z ffn +Z

[0089] Among them, ρ represents the Swish activation function, Z represents the output of the FCCE unit, and M ln represents the residual connection before calculating the block attention, that is, the fusion result of the spectrum block before layer normalization and the output of the FCCE unit, W T , represents the parameter matrix of the feedforward linear layer, and b, b1, b2 represent the bias terms of the feedforward linear layer.

[0090] In one embodiment, the cross entropy with smoothing factor is used to calculate the loss and back propagate to optimize the speech emotion recognition network parameters. A large number of experiments have shown that cross entropy is prone to overfitting due to factors such as data imbalance during training. Therefore, the present invention introduces a label smoothing strategy with a smoothing factor of 0.1 in the loss calculation process. The specific calculation is as follows: d Get Z through global average pooling (GAP) p , and then use the Softmax function to normalize the probability distribution:

[0091] Z p =(GAP(Z d )W T +b)

[0092]

[0093] Among them, q i Represents the predicted probability of the i-th category. i It represents the result after global average pooling, that is, the probability that the current data belongs to the i-th category, and then the cross entropy calculation with smoothing factor α is performed:

[0094]

[0095]

[0096] Among them, K represents the total number of emotion categories, α is set to 0.1, and target represents the original label of the current data. The loss L is used for back-propagation optimization of model parameters. During the training process, data verification is performed every 3 epochs, and the model parameters with the highest accuracy of the verification set are saved.

[0097] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0098] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A noisy speech emotion recognition method based on deep learning, characterized in that: The following steps are involved: S1. Acquire audio data and preprocess it to obtain preprocessed data; S2. Extract the Mel-spectrogram features and time-frequency features of the preprocessed data, and randomly mask the frequency domain and time domain of the Mel-spectrogram features to obtain random masking features; S3. Construct a speech emotion recognition network, and use the hybrid features obtained by combining time-frequency features and random masking features to train the speech emotion recognition network; the speech emotion recognition network includes a time-frequency embedding module, an absolute position encoding module, a hybrid feature map block module and a backbone model, the backbone model is stacked by multiple repeated modules with the same structure, the input of each repeated module is dynamically changed, and the repeated modules include a layer normalization unit, a multi-head block attention calculation unit, a frequency domain compression and channel enhancement unit, a downsampling and channel expansion module and a feedforward neural network unit; S4. Use the trained speech emotion recognition network to recognize the speech information to be tested.

2. The method for emotion recognition of noisy speech based on deep learning according to claim 1, characterized in that: The process of obtaining mixed features is: Extract the Mel-spectrogram features of the preprocessed data, expressed as: M=log(melW·(STFT(X)) 2 ) Randomly masking the frequency domain and time domain of the Mel-spectrogram feature obtains the random masking feature, which is expressed as: M′=Specaugment(M) The time-frequency features of the preprocessed data are extracted using stacked residual one-dimensional convolution, expressed as: N = StackedConv1D(X) The time-frequency feature is combined with the random masking feature to obtain the hybrid feature, which is expressed as: F = concat([M′,N], dim = 1) Where X∈R (B,T) represents preprocessed data, represents fast Fourier transform, melW represents Mel filter bank, represents the data augmentation operation of randomly masking the time domain and frequency domain, represents the residual one-dimensional convolution, represents the splicing combination, M represents the Mel-spectrogram feature, M ′ represents random masking features, N represents time-frequency features, and F represents mixed features.

3. The method for emotion recognition of noisy speech based on deep learning according to claim 1, characterized in that: The processing process of the frequency domain compression and channel enhancement unit is: Compress the output of the multi-head attention unit in the frequency domain to obtain compressed features; The compressed features are self-attentioned frame by frame, and the output of the frame-by-frame self-attention calculation is channel-enhanced with a smoothing factor.

4. The method for emotion recognition of noisy speech based on deep learning according to claim 1, characterized in that: The training process of the speech emotion recognition network includes: S11. Use two-dimensional convolution to embed the mixed features into time and frequency to obtain high-dimensional features; S12. Absolute position encoding is performed on each audio frame in the high-dimensional feature; S13. Divide the high-dimensional features after absolute position encoding into blocks to obtain multiple spectrum blocks; S14. Input all spectrum blocks into the backbone model for calculation until all spectrum blocks are calculated; S15. After performing global average pooling on the calculation results of S14, the cross entropy with a smoothing factor is used to calculate the loss and back propagate to optimize the parameters of the speech emotion recognition network.

5. The method for noisy speech emotion recognition based on deep learning according to claim 4, characterized in that: The cross entropy with smoothing factor is expressed as: Among them, q i represents the predicted probability of the i-th category, K represents the total number of sentiment categories, α represents the smoothing factor, target represents the original label, and z i Represents the result after global average pooling.

Images