A speech emotion recognition method and system based on time-frequency feature double fusion

Abstract

The application provides a speech emotion recognition method and system based on time-frequency feature double fusion, relates to the field of speech emotion recognition, and comprises the following steps: obtaining a speech signal to be recognized and performing pretreatment to obtain a mel-frequency cepstral coefficient of the speech signal; extracting a deep time feature and a deep frequency feature of the mel-frequency cepstral coefficient; adaptively fusing and globally fusing the deep time feature and the deep frequency feature to obtain a time-frequency feature; reshaping the obtained time-frequency feature, feeding the reshaped time-frequency feature into a full connection network for classification, and determining an emotion corresponding to the speech signal according to a classification result. The application can fully utilize emotional information features contained in the speech signal and improve the accuracy of speech emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of speech emotion recognition, and specifically to a speech emotion recognition method and system based on dual fusion of time and frequency features. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Human emotional states are a crucial factor in human communication, and speech plays a vital role in emotional expression. Speech emotion recognition (SER) aims to predict the emotions reflected in speech. Due to its widespread applications in human-computer interaction, social robotics, healthcare, and smart homes, SER has attracted significant attention from both industry and academia. SER can be viewed as a classification task with discrete emotion labels, such as sadness, anger, and happiness. It first extracts features from speech samples and then classifies the samples into one of several emotion labels.

[0004] The core technology of speech emotion recognition is feature extraction, which involves extracting representative features from the speech signal to distinguish different emotional states. Existing technologies typically use handcrafted acoustic features (LLDs) of speech, such as prosodic features (e.g., pitch, intensity, and duration) and spectral features (e.g., linear prediction coefficients (LPC) and linear prediction cepstral coefficients (LPCC)). However, speech is a time-varying signal; the spectral distribution of the speech signal changes over time. The aforementioned features can only simply describe the speech characteristics and do not fully extract the emotional information contained within the speech signal. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a speech emotion recognition method and system based on dual fusion of time and frequency features, so as to fully utilize the emotional information features contained in the speech signal and improve the accuracy of speech emotion recognition.

[0006] To achieve the above objectives, according to some embodiments, a first aspect of the present invention provides a speech emotion recognition method based on dual fusion of time and frequency features, comprising:

[0007] The speech signal to be recognized is acquired and preprocessed to obtain the Mel-frequency cepstral coefficients of the speech signal;

[0008] Extract the depth-time and depth-frequency features of the Mel frequency cepstral coefficients;

[0009] The time-frequency features are obtained by adaptively fusing and globally fusing the depth-time features and depth-frequency features;

[0010] The obtained time-frequency features are reshaped and fed into a fully connected network for classification. The emotion corresponding to the speech signal is determined based on the classification results.

[0011] Preferably, the steps of acquiring the speech signal to be recognized and preprocessing it to obtain the Mel-frequency cepstral coefficients of the speech signal include:

[0012] The speech to be recognized is acquired, and the speech is segmented into frames and silence at the beginning and end is removed to obtain the speech signal.

[0013] After windowing the speech signal, a short-time Fourier transform is performed to obtain the power spectrum. The power spectrum is then input into a Mel filter bank, and after logarithmic operations, a log-Mel spectrum is obtained. Finally, a discrete cosine transform is applied to the log-Mel coefficients to obtain the Mel frequency cepstral coefficients.

[0014] Preferably, the step of extracting the depth temporal features includes:

[0015] Shallow temporal features of Mel frequency cepstral coefficients are extracted using two-dimensional convolution;

[0016] The shallow time features are transposed, and the transposed time features are then subjected to global average pooling to obtain the time mapping matrix.

[0017] The obtained time mapping matrix is ​​input into two consecutive fully connected layers to obtain the time weights;

[0018] The obtained time weights are multiplied and weighted element-wise with the transposed time features. The weighted result is then transposed to obtain the deep time features.

[0019] Preferably, the depth frequency feature extraction step includes:

[0020] Shallow frequency features of Mel frequency cepstral coefficients are extracted using two-dimensional convolution;

[0021] The shallow frequency features are transposed, and the transposed frequency features are then subjected to global average pooling to obtain the frequency mapping matrix.

[0022] The obtained frequency mapping matrix is ​​input into two consecutive fully connected layers to obtain the frequency weights;

[0023] The obtained frequency weights are multiplied and weighted element by element with the transposed frequency features. The weighted result is then transposed to obtain the deep frequency features.

[0024] Preferably, the adaptive fusion and global fusion of depth temporal features and depth frequency features to obtain time-frequency features specifically includes the following steps:

[0025] The depth time features and depth frequency features are stitched together on the channel to obtain the stitched features;

[0026] The concatenated features are subjected to a two-dimensional convolution operation to generate a weight map. The weight map is then normalized to obtain the spatial importance weights.

[0027] Based on spatial importance weights, depth temporal features and depth frequency features are weighted and fused to obtain fused features;

[0028] Flatten the fused features to obtain flattened fused features, and then perform global fusion on the obtained flattened fused features to obtain time-frequency features.

[0029] Preferably, the step of globally fusing the obtained flattened fusion features to obtain time-frequency features includes:

[0030] The flattened fusion features are then normalized, fully connected, and activated using GeLU to obtain activated features.

[0031] The activation feature is divided into two segmentation features along the feature dimension. One segmentation feature is standardized and subjected to a two-dimensional convolution operation, and then multiplied by the other segmentation feature to obtain a cross-dimensional fusion feature.

[0032] The time-frequency features are obtained by residually connecting the cross-dimensional fusion features and the flattened fusion features.

[0033] Preferably, the step of reshaping the obtained time-frequency features and feeding them into a fully connected network for classification, and determining the emotion corresponding to the speech signal based on the classification result, specifically includes:

[0034] The obtained time-frequency features are reshaped and fed into a fully connected network. The output is then processed through a Softmax network layer for speech emotion classification.

[0035] Cross-entropy loss is calculated using the output features of a fully connected network and real sentiment labels.

[0036] A second aspect of the present invention provides a speech emotion recognition system based on dual fusion of time and frequency features, comprising:

[0037] The preprocessing module is configured to: acquire the speech signal to be recognized and preprocess it to obtain the Mel-frequency cepstral coefficients of the speech signal;

[0038] The feature extraction module is configured to extract depth-time features and depth-frequency features of Mel-frequency cepstral coefficients;

[0039] The feature fusion module is configured to adaptively fuse and globally fuse depth temporal features and depth frequency features to obtain time-frequency features.

[0040] The emotion recognition module is configured to: reshape the obtained time-frequency features, feed them into a fully connected network for classification, and determine the emotion corresponding to the speech signal based on the classification results.

[0041] A third aspect of the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps of the above-described speech emotion recognition method based on dual fusion of time and frequency features.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the above-described speech emotion recognition method based on dual fusion of time and frequency features.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] This invention discloses a speech emotion recognition method and system based on dual fusion of time and frequency features. It extracts shallow time and frequency features from Mel-frequency cepstral coefficients, then processes these shallow features through a time and frequency feature extraction module. Important feature weights are increased, while useless feature weights are decreased, helping the network extract deep time and frequency features rich in emotion from the shallow features. Adaptive feature fusion is employed to more fully fuse the extracted time and frequency features. Following adaptive feature fusion, a global fusion module is added to further extract emotion-rich features globally.

[0045] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0046] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0047] Figure 1 A flowchart of the speech emotion recognition method provided in the embodiments of the present invention;

[0048] Figure 2 This is a schematic diagram of the internal structure of the time and frequency feature extraction module provided in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the internal structure of the adaptive fusion and global fusion modules provided in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the structure of a voice emotion recognition system provided in an embodiment of the present invention. Detailed Implementation

[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0052] Example 1

[0053] like Figure 1 As shown, Embodiment 1 of the present invention provides a speech emotion recognition method based on dual fusion of time and frequency features, comprising:

[0054] S1. Acquire the speech signal to be recognized and preprocess it to obtain the Mel frequency cepstral coefficients of the speech signal;

[0055] S2. Extract the depth-time and depth-frequency features of the Mel frequency cepstral coefficients;

[0056] S3. Adaptively fuse the depth time features and the depth frequency features and perform global fusion to obtain the time-frequency features;

[0057] S4. The obtained time-frequency features are reshaped and fed into a fully connected network for classification. The emotion corresponding to the speech signal is determined based on the classification results.

[0058] In step S1, the speech signal to be recognized is acquired and preprocessed to obtain the Mel-frequency cepstral coefficients of the speech signal, specifically including:

[0059] Step S11: Obtain the speech to be recognized, divide the speech into frames, remove the first and last silences, and unify the length of the speech to obtain the speech signal.

[0060] The .wav format audio files in the dataset were preprocessed. Using an initial sampling frequency of 16000Hz, noise reduction and removal of beginning and ending silences were performed. Then, based on experimental results, a uniform length of 2 seconds was selected for each audio segment. Segments longer than 2 seconds were trimmed, and segments shorter than 2 seconds were padded. This divided the audio signal into small, frame-by-frame segments for easier analysis.

[0061] Step S12: Window the speech signal, perform short-time Fourier transform on the windowed speech signal to obtain the power spectrum, input the power spectrum into the Mel filter bank, perform logarithmic operation to obtain the Log-Mel spectrum, and apply discrete cosine transform to the logarithmic Mel coefficients to obtain the Mel frequency cepstral coefficients.

[0062] To ensure smooth transitions between frames and maintain continuity, the speech signal is windowed. Specifically, a Hamming window is used to window the speech signal, and the Hamming window formula is as follows:

[0063]

[0064] Where W(n) is the window function and N is the number of sampling points per frame.

[0065] The windowed speech signal is subjected to a Fast Fourier Transform (FFT) frame by frame to obtain the spectrum of each frame. The power spectral density (PSD) is obtained by taking the modulus and square of the spectrum, thus converting the time-domain signal into an energy distribution in the frequency domain. Further, the power spectrum is input into a Mel filter bank, using a triangular filter configuration. More dense filters are used for frequencies closer to human hearing, while fewer filters are used for higher frequencies. The logarithm of the energy output of each Mel filter is taken to obtain the Log-Mel spectrum. A Discrete Cosine Transform (DCT) is applied to the logarithmic Mel coefficients to obtain the Mel-frequency cepstral coefficients (MFCCs).

[0066] In step S2, the depth-time features and depth-frequency features of the Mel-frequency cepstral coefficients are extracted. A time feature extraction module and a frequency feature extraction module are used to extract the depth-time features and depth-frequency features from the Mel-frequency cepstral coefficients, respectively. The time feature extraction module and the frequency feature extraction module each include a two-dimensional convolutional layer, a global average pooling layer, two fully connected layers, and two transposed layers, as shown below. Figure 2 As shown, the kernel size of the two-dimensional convolutional layer in the temporal feature extraction module is 1×3, the kernel size of the two-dimensional convolutional layer in the frequency feature extraction module is 3×1, and the structures of other layers are the same.

[0067] The specific extraction steps are as follows:

[0068] Step S21: Using two-dimensional convolution, by setting different convolution kernel sizes, shallow temporal features X of Mel frequency cepstral coefficients are extracted. t ∈R C×H×W and frequency characteristics X f ∈R C×H×W .

[0069] Step S22, X t and X f Perform a transpose operation, swapping the channel and time / frequency dimensions respectively, so that X t The dimensional order is converted to X t ∈R W×H×C X f The dimensional order is converted to X f ∈R H×C×W ; For Xt and X f Global average pooling is used to compress the channel and frequency dimensions, and the channel and time dimensions respectively, generating two mappings P. t ∈R W×1×1 P f ∈R H×1×1 ;

[0070] Step S23: To learn the saliency of temporal and frequency features, two fully connected layers with ReLU and Tanh activation functions are used to learn temporal weights. The fully connected layers are used to better fuse all input feature information. The specific implementation process is as follows:

[0071] Z t =σ(δ(P) t W t1 W t2 )∈R W×1×1

[0072] Z f =σ(δ(P) f W f1 W f2 )∈R H×1×1

[0073] Where σ() is the Tanh activation function, δ() is the ReLU activation function, and W * t1, t2, f1, and f2 are the weight matrices of the fully connected layers to be trained, and t1, t2, f1, and f2 are the indices of different fully connected networks in the feature extraction module.

[0074] Step S24: The time weights and frequency weights obtained through the above operations are multiplied element-wise and applied to the original features, thereby redistributing the original features in the time and frequency dimensions. The calculation formula is as follows:

[0075]

[0076]

[0077] In the formula, This represents element-wise multiplication.

[0078] Step S25, finally X′ t and X′ f The channels are exchanged with the time and frequency dimensions to obtain the depth time feature X. T ∈R C×H×W and depth frequency features X F ∈R C×H×W .

[0079] In step S3, the depth temporal features and depth frequency features are adaptively fused and globally fused to obtain time-frequency features. The specific steps are as follows: Figure 3 As shown, it includes:

[0080] Step S31: For the depth-time feature X obtained in S2 T ∈R C×H×W and depth frequency features X F ∈R C×H×W , will X T and X F The splicing feature is obtained by splicing along the channel direction.

[0081]

[0082] Step S32: Generate a weight map W through convolution operation. TF ∈R 2×H×W The Softmax operator is used to normalize the weight map to obtain weighting coefficients, which represent the importance of features at different locations.

[0083]

[0084] The split operation splits the weight graph into α. T ∈R 1×H×W β F ∈R 1×H×W α T β F The spatial importance weights corresponding to the time and frequency feature maps are shared across all channels.

[0085] Step S33: Weight the input feature maps according to spatial importance weights, so that the depth temporal features and depth frequency features are adaptively fused in space to obtain the fused feature X. TF :

[0086] X TF =α T X T +β F X F ∈R C×H×W

[0087] Step S34: For feature X TF Perform flattening feature mapping to obtain Where D f =H×W; then perform global fusion. The global fusion module consists of two fully connected layers, one 2D convolutional layer, two normalization layers, one GeLU activation layer, and one multiplication operation; first, after normalization, fully connected layers, and GeLU activation, we get:

[0088]

[0089] Where Norm() represents normalization, W g1 Let ξ be the weight matrix of the fully connected layer, and let ξ() represent the GeLU activation function.

[0090] Step S35: After projection through the GeLU activation function, the output X is segmented along the feature dimension into... That is, X = concat(Z1, Z2); Z2 is standardized and then subjected to a 1*1 2D convolution, and then multiplied by Z1 to enhance cross-dimensional feature fusion, i.e.:

[0091]

[0092] Where ⊙ represents matrix dot product, and S is a cross-dimensional fusion feature;

[0093] Step S36: Finally, residual connections are used to further extract sentiment features globally, resulting in time-frequency features X. g :

[0094]

[0095] Among them, W g2 This is the weight matrix of the fully connected layer.

[0096] Step S4 involves reshaping the obtained time-frequency features, feeding them into a fully connected network for classification, and determining the emotion corresponding to the speech signal based on the classification result. Specific steps include:

[0097] The time-frequency features generated after dual feature fusion are reshaped and fed into a fully connected network, transforming the output into a 4-dimensional vector. Reshaping reduces the dimensionality of the time-frequency features, facilitating input into the fully connected network. The output then passes through a Softmax network layer for speech emotion classification. The Softmax network layer is a general network structure that represents the output in the [0, 1] interval, finding the dimension with the highest probability as the final output, i.e., the final emotion.

[0098] y i =Softmax(FC(X) g ))

[0099] Here, FC() represents a fully connected operation.

[0100] Example 2

[0101] like Figure 4 As shown, this embodiment provides a speech emotion recognition system based on dual fusion of time and frequency features, including:

[0102] The preprocessing module is configured to: acquire the speech signal to be recognized and preprocess it to obtain the Mel-frequency cepstral coefficients of the speech signal;

[0103] The feature extraction module is configured to extract depth-time features and depth-frequency features of Mel-frequency cepstral coefficients;

[0104] The feature fusion module is configured to adaptively fuse and globally fuse depth temporal features and depth frequency features to obtain time-frequency features.

[0105] The emotion recognition module is configured to: reshape the obtained time-frequency features, feed them into a fully connected network for classification, and determine the emotion corresponding to the speech signal based on the classification results.

[0106] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0107] Example 3

[0108] This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it completes the steps of the speech emotion recognition method based on dual fusion of time and frequency features in Embodiment 1.

[0109] Example 4

[0110] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, they complete the steps of the speech emotion recognition method based on dual fusion of time and frequency features in Embodiment 1.

[0111] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A speech emotion recognition method based on dual fusion of time and frequency features, characterized in that, include: The speech signal to be recognized is acquired and preprocessed to obtain the Mel-frequency cepstral coefficients of the speech signal; Extract the depth-time and depth-frequency features of the Mel frequency cepstral coefficients; The time-frequency features are adaptively and globally fused with the depth-time features and depth-frequency features. The specific steps include: The depth time features and depth frequency features are stitched together on the channel to obtain the stitched features; The concatenated features are subjected to a two-dimensional convolution operation to generate a weight map. The weight map is then normalized to obtain the spatial importance weights. Based on spatial importance weights, depth temporal features and depth frequency features are weighted and fused to obtain fused features; Flatten the fused features by performing feature mapping to obtain flattened fused features; The flattened fusion features are then normalized, fully connected, and activated using GeLU to obtain activated features. The activation feature is divided into two segmentation features along the feature dimension. One segmentation feature is standardized and subjected to a two-dimensional convolution operation, and then multiplied by the other segmentation feature to obtain a cross-dimensional fusion feature. By performing residual connection between the cross-dimensional fusion features and the flattened fusion features, time-frequency features are obtained; The obtained time-frequency features are reshaped and fed into a fully connected network for classification. The emotion corresponding to the speech signal is determined based on the classification results.

2. The speech emotion recognition method based on dual fusion of time and frequency features as described in claim 1, characterized in that, The steps for acquiring the speech signal to be recognized and preprocessing it to obtain the Mel-frequency cepstral coefficients of the speech signal include: The speech to be recognized is obtained, the beginning and end silences of the speech are removed and the speech is divided into frames to obtain the speech signal; After windowing the speech signal, a short-time Fourier transform is performed to obtain the power spectrum. The power spectrum is then input into a Mel filter bank, and logarithmic operations are performed to obtain the logarithmic Mel coefficients. Finally, a discrete cosine transform is applied to the logarithmic Mel coefficients to obtain the Mel frequency cepstral coefficients.

3. The speech emotion recognition method based on dual fusion of time and frequency features as described in claim 1, characterized in that, The steps for extracting the depth temporal features include: Shallow temporal features of Mel frequency cepstral coefficients are extracted using two-dimensional convolution; The shallow time features are transposed, and the transposed time features are then subjected to global average pooling to obtain the time mapping matrix. The obtained time mapping matrix is ​​input into two consecutive fully connected layers to obtain the time weights; The obtained time weights are multiplied and weighted element-wise with the transposed time features. The weighted result is then transposed to obtain the deep time features.

4. The speech emotion recognition method based on dual fusion of time and frequency features as described in claim 1, characterized in that, The steps for extracting the depth frequency features include: Shallow frequency features of Mel frequency cepstral coefficients are extracted using two-dimensional convolution; The shallow frequency features are transposed, and the transposed frequency features are then subjected to global average pooling to obtain the frequency mapping matrix. The obtained frequency mapping matrix is ​​input into two consecutive fully connected layers to obtain the frequency weights; The obtained frequency weights are multiplied and weighted element by element with the transposed frequency features. The weighted result is then transposed to obtain the deep frequency features.

5. The speech emotion recognition method based on dual fusion of time and frequency features as described in claim 1, characterized in that, The process of reshaping the obtained time-frequency features and feeding them into a fully connected network for classification, and determining the emotion corresponding to the speech signal based on the classification result, includes the following steps: The obtained time-frequency features are reshaped and fed into a fully connected network. The output of the fully connected network is then passed through a Softmax layer for speech emotion classification. Cross-entropy loss is calculated using the output features of a fully connected network and real sentiment labels.

6. A speech emotion recognition system based on dual fusion of time and frequency features, characterized in that, include: The preprocessing module is configured to: acquire the speech signal to be recognized and preprocess it to obtain the Mel-frequency cepstral coefficients of the speech signal; The feature extraction module is configured to extract depth-time features and depth-frequency features of Mel-frequency cepstral coefficients; The feature fusion module is configured to adaptively and globally fuse depth temporal features and depth frequency features to obtain time-frequency features. Specific steps include: The depth time features and depth frequency features are stitched together on the channel to obtain the stitched features; The concatenated features are subjected to a two-dimensional convolution operation to generate a weight map. The weight map is then normalized to obtain the spatial importance weights. Based on spatial importance weights, depth temporal features and depth frequency features are weighted and fused to obtain fused features; Flatten the fused features by performing feature mapping to obtain flattened fused features; The flattened fusion features are then normalized, fully connected, and activated using GeLU to obtain activated features. The activation feature is divided into two segmentation features along the feature dimension. One segmentation feature is standardized and subjected to a two-dimensional convolution operation, and then multiplied by the other segmentation feature to obtain a cross-dimensional fusion feature. By performing residual connection between the cross-dimensional fusion features and the flattened fusion features, time-frequency features are obtained; The emotion recognition module is configured to: reshape the obtained time-frequency features, feed them into a fully connected network for classification, and determine the emotion corresponding to the speech signal based on the classification results.

7. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps of the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the steps of the method according to any one of claims 1-5.

Images