Speech emotion recognition method and device fusing multi-level acoustic information and storage medium

Abstract

The application discloses a speech emotion recognition method and device fusing multi-level acoustic information and a storage medium, and the method comprises the following steps: S01, obtaining a preprocessed speech signal to be recognized to obtain a spectrum feature map and a mel-frequency cepstrum coefficient; S02, respectively extracting a spectrum feature, an MFCC feature, and a W2E feature; S03, inputting the spectrum feature, the MFCC feature, and the W2E feature into a cross-level attention network to be fused to obtain a fusion feature output, the cross-level attention network comprising three groups of cross-level attention modules, each group being used for fusing two features by using self-attention, and the three groups of fusion features being spliced to obtain a final fusion feature; and S04, inputting the fusion feature into a classifier to obtain an emotion classification result. The application can fuse acoustic features of different levels by using a cross-level attention mode, and improve the recognition accuracy and the anti-interference performance of emotion recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech emotion recognition, and in particular to a speech emotion recognition method and device fusing multi-level acoustic information and a storage medium. BACKGROUND

[0002] Speech, as an acoustic representation of language, contains rich linguistic and non-linguistic emotional information in addition to expressing the speaker's intention. Intelligent speech interaction systems enable machines to understand the semantic information of the speaker, but there is still a great challenge in making machines understand the emotions of the speaker.

[0003] The existing speech emotion recognition method usually preprocesses the original speech first, and then extracts some features from the original speech to identify the emotion category to which the features belong. However, the distinguishing ability of this type of recognition method is limited, for example, the recognition accuracy for specific category emotions is not high. Neural network models can extract higher-level and more rich features from spectrograms, thereby improving recognition accuracy and effectively solving the above problems. However, the features that can be extracted from spectrograms are limited, so it is usually necessary to rely on multi-modal emotion recognition. For example, acoustic information and text information are combined, and context information is learned to help make the final emotion prediction. However, it may be difficult to obtain multi-modal emotion information in actual applications, resulting in multi-modal network models that are not applicable, and only single-modal network models can be used, and multi-modal emotion recognition also has the following problems:

[0004] 1. prone to execution errors. Multi-modal data-based recognition is prone to errors that interfere with the normal execution of the emotion recognition task, such as text transcription errors.

[0005] 2. Lack of feature correlation information. Most methods based on CNN or LSTM, attention mechanism can only extract one of the spatial information or time sequence information of the speech, and do not fully utilize multiple feature information, lack of correlation information between different feature information, so that the emotion recognition accuracy still needs to be improved. SUMMARY

[0006] The technical problem to be solved by the present application is that in view of the technical problems existing in the prior art, the present application provides a speech emotion recognition method, device and storage medium fusing multi-level acoustic information, which is simple to implement, low in cost, high in recognition accuracy, and strong in anti-interference.

[0007] To solve the above technical problems, the technical solution provided by the present application is:

[0008] A speech emotion recognition method fusing multi-level acoustic information, comprising the steps of:

[0009] S01. Preprocessing: Acquire the speech signal to be recognized and preprocess it. Obtain the spectral feature map and Mel-frequency cepstral coefficients based on the preprocessed signal.

[0010] S02. Feature extraction: Extract spectral features and MFCC features from the spectral feature map and Mel-frequency cepstral coefficients, respectively, and extract W2E (wav2vec2) features from the speech signal to be processed;

[0011] S03. Cross-attention fusion: The spectral features, MFCC features, and W2E features are input into a cross-level attention network for fusion to obtain a fused feature output. The cross-level attention network includes three sets of cross-level attention modules. Each set of cross-level attention modules is used to fuse two of the spectral features, MFCC features, and W2E features using self-attention to obtain a fused feature of the two features. The fused features obtained from the three sets of cross-level attention modules are spliced ​​together to obtain the final fused feature.

[0012] S04. Sentiment Classification: Input the fused features into a pre-trained classifier to obtain the sentiment classification result.

[0013] Furthermore, step S01 includes:

[0014] S101. The speech signal to be recognized is segmented into frames, silences at the beginning and end are removed, and the speech length is unified to obtain the preprocessed signal;

[0015] S102. Window the preprocessed signal, perform a short-time Fourier transform on the windowed signal to obtain the power spectrum, perform logarithmic operations on the power spectrum to obtain the spectral feature map, and input the power spectrum into a Mel filter bank to obtain the Mel cepstral coefficients through logarithmic operations and discrete cosine transform.

[0016] Furthermore, in step S102, the preprocessed signal is windowed using the Hamming window shown below:

[0017]

[0018] Where N is the number of frames and n is the current frame number.

[0019] Furthermore, in step S02, a two-dimensional convolutional neural network is used to learn spectral features from the spectral feature map, a long short-term memory network is used to learn MFCC features from the spectral feature map, and a pre-trained model is used to extract the W2E features.

[0020] Furthermore, each of the cross-level attention modules includes three LN normalization layers, one multi-head attention layer, and one position connection layer. The first LN normalization layer and the second LN normalization layer respectively input any two of the spectral features, MFCC features, and W2E features, and after processing, output them to the multi-head attention layer. The output of the multi-head attention layer and the output of the first LN normalization layer are added to obtain a first accumulation result, which is then output to the third LN normalization layer. The output of the third LN normalization layer is added to the first accumulation result after passing through the position connection layer to obtain the fusion result between the two input features.

[0021] Furthermore, the calculation expression for the cross-level attention module is as follows:

[0022]

[0023]

[0024] Where α represents any one of the spectral features, MFCC features, and W2E features, β represents one of the other two features besides α, and f θ It is a positionally fully connected layer with θ-parameterization. LN represents multi-head attention from β to α, and LN represents layer normalization.

[0025] Furthermore, in step S04, two softmax classifiers are used to input the fused features obtained in step S03 into the corresponding classification networks to classify speech emotion and speaker gender, respectively. The loss of the classification network is calculated using the following formula:

[0026] L=αL CE-emo +(1-α)L CE-gen

[0027] Among them, L CE-emo It is the cross-entropy loss function for the sentiment classification task, L CE-gen α is the cross-entropy loss function for the gender classification task, and α is the weight coefficient of the emotion recognition loss function. The entire network is backpropagated and gradients are updated using the joint loss L.

[0028] A voice emotion recognition device that integrates multi-level acoustic information includes:

[0029] The preprocessing module is used to acquire the speech signal to be recognized, perform preprocessing, and obtain the spectral feature map and Mel-frequency cepstral coefficients based on the preprocessed signal.

[0030] The feature extraction module is used to extract spectral features and MFCC features from the spectral feature map and Mel-frequency cepstral coefficients, respectively, and to extract W2E features from the speech signal to be processed.

[0031] The cross-attention fusion module is used to input the spectral features, MFCC features and W2E features into the cross-level attention network for fusion to obtain the fused feature output. The cross-level attention network includes three sets of cross-level attention modules. Each set of cross-level attention modules is used to fuse two of the spectral features, MFCC features and W2E features using self-attention to obtain the fused features of the two features. The fused features obtained from the three sets of cross-level attention modules are concatenated to obtain the final fused feature.

[0032] The sentiment classification module is used to input the fused features into a pre-trained classifier to obtain the sentiment classification result.

[0033] A computer device includes a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to perform the method described above.

[0034] A computer-readable storage medium storing a computer program that, when executed, implements the method described above.

[0035] Compared with existing technologies, the advantages of this invention are as follows: By extracting three different levels of acoustic features—spectral features, MFCC features, and W2E features—this invention can reduce the loss of global features. Simultaneously, by using cross-level attention, the three features are fused pairwise, allowing features at different levels to adapt to each other and establish connections. This enables the effective fusion of high-level and low-level emotional features, fully utilizing the emotional features extracted from speech signals to learn the relationships between different levels of features. This results in richer and more diverse features, forming deeper features that effectively fuse important information from each level. Furthermore, it allows for the acquisition of connections between features at different levels, effectively improving the accuracy and anti-interference capabilities of emotion recognition. It is also less prone to errors that interfere with the normal execution of the emotion recognition task, leading to a higher success rate. Attached Figure Description

[0036] Figure 1 This is a schematic diagram illustrating the implementation process of the speech emotion recognition method that integrates multi-level acoustic information in this embodiment.

[0037] Figure 2 This is a schematic diagram illustrating the structural principle of speech emotion recognition that integrates multi-level acoustic information in this embodiment.

[0038] Figure 3 This is a schematic diagram illustrating the structural principle of the cross-level attention module in this embodiment.

[0039] Figure 4 This is a schematic diagram of the principle of the voice emotion recognition device that integrates multi-level acoustic information in this embodiment. Detailed Implementation

[0040] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0041] As disclosed in this invention, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. The terms "first," "second," and similar terms used in this invention disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, words such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects.

[0042] like Figure 1 As shown, the steps of the speech emotion recognition method that integrates multi-level acoustic information in this embodiment include:

[0043] S01. Preprocessing: Acquire the speech signal to be recognized and preprocess it. Obtain the spectral feature map and Mel-frequency cepstral coefficients based on the preprocessed signal.

[0044] In this embodiment, the speech to be recognized is first acquired. The acquired speech is then preprocessed by methods such as frame segmentation, removal of beginning and end silences, and standardization of speech length to obtain a preprocessed signal. Based on the preprocessed signal, after windowing and short-time Fourier transform, the spectral feature map and Mel-frequency cepstral coefficients are obtained.

[0045] In this embodiment, the detailed steps for acquiring the speech signal to be recognized, performing preprocessing, and obtaining the spectral feature map and Mel-frequency cepstral coefficients based on the preprocessed signal are as follows:

[0046] Step S101: Obtain the speech to be recognized, and perform preprocessing on the obtained speech by methods such as frame segmentation, removal of first and last silences, and standardization of speech length to obtain the speech signal.

[0047] In a specific application embodiment, the .wav format audio files in the dataset to be processed are preprocessed. Using an initial sampling frequency of 16kHz, the audio is segmented into frames, and initial and final silences are removed. A uniform length of 3 seconds is selected for each audio segment; segments longer than 3 seconds are trimmed, and segments shorter than 3 seconds are filled with silence. The audio is then divided into 3-second frames to facilitate subsequent analysis of the audio signal.

[0048] Step S102: Window the speech signal, perform short-time Fourier transform on the windowed speech signal to obtain the power spectrum, perform logarithmic operation on the power spectrum to obtain the spectral feature map; input the power spectrum into the Mel filter bank, and after logarithmic operation and discrete cosine transform, obtain the Mel cepstral coefficients.

[0049] In a specific application embodiment, step S102 above can be performed by windowing the preprocessed signal using the following Hamming window:

[0050]

[0051] Where N is the number of frames and n is the current frame number.

[0052] S02. Feature Extraction: Extract spectral features and MFCC features from the spectral feature map and Mel-Cepstral Coefficients, respectively, and extract W2E features from the speech signal to be processed.

[0053] This embodiment specifically uses a two-dimensional convolutional neural network to learn spectral features from the spectral feature map, a long short-term memory network (LSTM) to learn MFCC features from Mel-frequency cepstral coefficients, and a pre-trained model to extract W2E features. In a specific application embodiment, the WAV2VEC2.0 pre-trained model can be used to extract W2E features from the speech signal. MFCC features are low-level features that take into account human auditory characteristics, while the spectral feature map and W2E are deep features of the audio signal. Combining spectral features, MFCC features, and W2E features can form multi-level acoustic information in the time and frequency domains.

[0054] like Figure 2 As shown, this embodiment extracts spectral features, MFCC features, and W2E features from the spectral feature map, Mel-Cepstral coefficients, and preprocessed speech data respectively by setting up a spectral feature extraction module, an MFCC feature extraction module, and a W2E feature extraction module.

[0055] S03. Cross-attention fusion: The spectral features, MFCC features, and W2E features are input into a cross-level attention network for fusion to obtain the fused feature output. The cross-level attention network includes three sets of cross-level attention modules. Each set of cross-level attention modules is used to fuse two of the spectral features, MFCC features, and W2E features using self-attention to obtain the fused features of the two features. The fused features obtained from the three sets of cross-level attention modules are concatenated to obtain the final fused feature.

[0056] In this embodiment, spectral features, MFCC features, and W2E features are input into a cross-level attention network to achieve the fusion of three emotion features. This cross-level attention network adopts three sets of cross-level attention modules (six modules). Each sub-module achieves adaptation from one level feature to another. Then, two sub-modules with the same target feature are fused using self-attention to obtain three sets of adapted features. Finally, the three sets of adapted features are concatenated to obtain fused features for classification, allowing two levels of features to adapt to the third level feature. By fusing the extracted spectral features, MFCC features, and W2E features across levels through cross-level attention, features at different levels are adapted to each other to obtain connections between them. This fully utilizes the features at each level of speech and the connections between them to enrich the feature dimensions, obtain deeper features, and enable the final features to effectively fuse important information from each level of features while obtaining the connections between features at each level. This solves the problem of single features in current methods, thereby effectively improving the accuracy of emotion recognition results.

[0057] like Figure 3 As shown, in this embodiment, each cross-level attention module specifically includes three LN normalization layers, one multi-head attention layer, and one positional connection layer. The first and second LN normalization layers respectively input any two of the following features: spectral features, MFCC features, and W2E features. After processing, these features are output to the multi-head attention layer. The output of the multi-head attention layer is added to the output of the first LN normalization layer to obtain a first accumulated result, which is then output to the third LN normalization layer. The output of the third LN normalization layer, after passing through the positional connection layer, is added to the first accumulated result to obtain the fusion result between the two input features. Based on the above structure, the specific calculation expression of the cross-level attention module is as follows:

[0058]

[0059]

[0060] Where α represents any one of the spectral features, MFCC features, and W2E features, β represents one of the other two features besides α, and f θ It is a positionally fully connected layer with θ-parameterization. LN represents multi-head attention from β to α, and LN represents layer normalization.

[0061] S04. Sentiment Classification: Input the fused features into a pre-trained classification network to obtain the sentiment classification result.

[0062] In this embodiment, two softmax classifiers can be used to classify speech emotion and speaker gender respectively. The fused features obtained in step S03 are input into the corresponding classification network for classification. Based on the classification results, the emotion and speaker gender corresponding to the speech signal are determined. This embodiment, based on emotion classification, adds an auxiliary task of speaker gender recognition. By sharing some model parameters, the model can learn the potential connection between speaker gender information and emotion information, thereby improving the model's generalization ability and further enhancing the accuracy of emotion recognition.

[0063] In a specific application embodiment, the loss of the classification network can be calculated using the following formula:

[0064] L=αL CE-emo +(1-α)L CE-gen (3)

[0065] Among them, L CE-emo It is the cross-entropy loss function for the sentiment classification task, L CE-gen α is the cross-entropy loss function for the gender classification task, and α is the weight coefficient of the emotion recognition loss function. The entire network is backpropagated and gradients are updated using the joint loss L.

[0066] This invention extracts three different levels of acoustic features—spectral features, MFCC features, and W2E features—to reduce the loss of global features. Furthermore, by employing cross-level attention, the three features are fused in pairs, achieving effective integration of high-level and low-level emotional features. This fully utilizes the emotional features extracted from speech signals at various levels and learns the relationships between features at different levels, resulting in richer and more diverse features. This effectively improves the accuracy and anti-interference capabilities of emotion recognition.

[0067] like Figure 4 As shown, the speech emotion recognition device that integrates multi-level acoustic information in this embodiment includes:

[0068] The preprocessing module is used to acquire the speech signal to be recognized, perform preprocessing, and obtain the spectral feature map and Mel-frequency cepstral coefficients based on the preprocessed signal.

[0069] The feature extraction module is used to extract spectral features, MFCC features, and W2E features from the spectral feature map, respectively;

[0070] The cross-attention fusion module is used to input spectral features, MFCC features, and W2E features into the cross-level attention network for fusion to obtain fused feature output. The cross-level attention network includes three sets of cross-level attention modules. Each set of cross-level attention modules is used to fuse two of the spectral features, MFCC features, and W2E features using self-attention to obtain fused features of the two features. The fused features obtained from the three sets of cross-level attention modules are concatenated to obtain the final fused feature.

[0071] The sentiment classification module is used to input fused features into a pre-trained classifier to obtain sentiment classification results.

[0072] The speech emotion recognition device that integrates multi-level acoustic information in this embodiment corresponds one-to-one with the speech emotion recognition method that integrates multi-level acoustic information described above, and will not be described in detail here.

[0073] This embodiment also provides a computer device, including a processor and a memory, the memory for storing a computer program and the processor for executing the computer program to perform the methods described above.

[0074] This embodiment also provides a computer-readable storage medium storing a computer program, which, when executed, implements the method described above.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A speech emotion recognition method that integrates multi-level acoustic information, characterized in that the steps include... include: S01. Preprocessing: Acquire the speech signal to be recognized and preprocess it. Obtain the spectral feature map and Mel-frequency cepstral coefficients based on the preprocessed signal. S02. Feature extraction: Extract spectral features and MFCC features from the spectral feature map and Mel-frequency cepstral coefficients, respectively, and extract W2E features from the speech signal to be processed; S03. Cross-attention fusion: The spectral features, MFCC features, and W2E features are input into a cross-level attention network for fusion to obtain a fused feature output. The cross-level attention network includes three sets of cross-level attention modules. Each set of cross-level attention modules is used to fuse two of the spectral features, MFCC features, and W2E features using self-attention to obtain a fused feature of the two features. The fused features obtained from the three sets of cross-level attention modules are spliced ​​together to obtain the final fused feature. S04. Sentiment Classification: Input the fused features into a pre-trained classifier to obtain the sentiment classification result.

2. The speech emotion recognition method integrating multi-level acoustic information according to claim 1, characterized in that, Step S01 includes: S101. The speech signal to be recognized is segmented into frames, silences at the beginning and end are removed, and the speech length is unified to obtain the preprocessed signal; S102. Window the preprocessed signal, perform a short-time Fourier transform on the windowed signal to obtain the power spectrum, perform logarithmic operations on the power spectrum to obtain the spectral feature map, and input the power spectrum into a Mel filter bank to obtain the Mel cepstral coefficients through logarithmic operations and discrete cosine transform.

3. The speech emotion recognition method integrating multi-level acoustic information according to claim 2, characterized in that, In step S102, the preprocessed signal is windowed using the Hamming window shown below: Where N is the number of frames and n is the current frame number.

4. The speech emotion recognition method integrating multi-level acoustic information according to claim 1, characterized in that, In step S02, a two-dimensional convolutional neural network is used to learn spectral features from the spectral feature map, a long short-term memory network is used to learn MFCC features from the spectral feature map, and a pre-trained model is used to extract the W2E features.

5. The speech emotion recognition method integrating multi-level acoustic information according to claim 1, characterized in that, Each of the aforementioned cross-level attention modules includes three LN normalization layers, one multi-head attention layer, and one positional connection layer. The first and second LN normalization layers respectively receive any two of the spectral features, MFCC features, and W2E features, and after processing, the results are output to the multi-head attention layer. The output of the multi-head attention layer and the output of the first LN normalization layer are added together to obtain a first accumulation result, which is then output to the third LN normalization layer. The output of the third LN normalization layer is then added to the first accumulation result after passing through the positional connection layer to obtain the fusion result between the two input features.

6. The speech emotion recognition method integrating multi-level acoustic information according to claim 5, characterized in that, The calculation expression for the cross-level attention module is as follows: Where α represents any one of the spectral features, MFCC features, and W2E features, β represents one of the other two features besides α, and f θ It is a positionally fully connected layer with θ-parameterization. LN represents multi-head attention from β to α, and LN represents layer normalization.

7. The speech emotion recognition method integrating multi-level acoustic information according to any one of claims 1 to 6, characterized in that, In step S04, two softmax classifiers are used to input the fused features obtained in step S03 into the corresponding classification networks to classify speech emotion and speaker gender, respectively. The loss of the classification network is calculated using the following formula: L=αL CE-emo +(1-α)L CE-gen Among them, L CE-emo It is the cross-entropy loss function for the sentiment classification task, L CE-gen α is the cross-entropy loss function for the gender classification task, and α is the weight coefficient of the emotion recognition loss function. The entire network is backpropagated and gradient updated through the joint loss L.

8. A voice emotion recognition device that integrates multi-level acoustic information, characterized in that, include: The preprocessing module is used to acquire the speech signal to be recognized, perform preprocessing, and obtain the spectral feature map and Mel-frequency cepstral coefficients based on the preprocessed signal. The feature extraction module is used to extract spectral features and MFCC features from the spectral feature map and Mel-frequency cepstral coefficients, respectively, and to extract W2E features from the speech signal to be processed. The cross-attention fusion module is used to input the spectral features, MFCC features and W2E features into the cross-level attention network for fusion to obtain the fused feature output. The cross-level attention network includes three sets of cross-level attention modules. Each set of cross-level attention modules is used to fuse two of the spectral features, MFCC features and W2E features using self-attention to obtain the fused features of the two features. The fused features obtained from the three sets of cross-level attention modules are concatenated to obtain the final fused feature. The sentiment classification module is used to input the fused features into a pre-trained classifier to obtain the sentiment classification result.

9. A computer device comprising a processor and a memory, the memory being used to store a computer program, characterized in that, The processor is used to execute the computer program to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed, it implements the method as described in any one of claims 1 to 7.

Images