Speech emotion recognition method and system based on spectrogram

Abstract

The invention discloses a speech emotion recognition method and system based on a spectrogram. The identification method comprises the following steps: acquiring an audio signal spectrogram, and extracting Mel-frequency cepstrum coefficient characteristics; performing emotion recognition according to the Mel-frequency cepstrum coefficient features by using an emotion classification model; the sentiment classification model sequentially comprises an input layer, a convolution layer, a pooling layer, a full connection layer, a Dropout layer and an output layer. According to the method, the spectrogram of the voice signal is analyzed through the deep learning technology, so that more accurate and more real-time emotion recognition is realized.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and more particularly to a speech emotion recognition method and system based on a spectrum graph. Background Art

[0002] Speech emotion recognition technology plays an important role in the field of artificial intelligence. It enables machines to understand and respond to human emotional states, thereby improving the naturalness and efficiency of human-computer interaction.

[0003] However, existing speech emotion recognition technologies generally rely on transcribing speech into text and then inferring emotions through text analysis, an approach that has multiple limitations.

[0004] First, it cannot directly extract emotional information from speech signals, thus ignoring non-verbal information such as the rhythm and pitch of speech, resulting in low accuracy in emotion recognition and intent parsing.

[0005] Secondly, there is insufficient support for multiple languages ​​and dialects. Accents and dialects from different regions and cultures have a significant impact on the accuracy of speech recognition. Existing speech emotion recognition methods are often optimized for specific languages ​​or dialects. For the recognition of other languages ​​or dialects, the differences in accents and pronunciations of different users lead to poor recognition results and low accuracy.

[0006] Finally, existing technologies have delay and lag issues when processing large-scale real-time voice data, especially in the process of converting speech to text, which affects the real-time response capability of speech recognition.

[0007] Therefore, it is particularly important to develop a method that can directly extract features from speech signals and perform emotion recognition. Summary of the invention

[0008] In view of this, the present invention provides a speech emotion recognition method and system based on spectrogram, which directly extracts emotion-related features from the spectrogram through deep learning technology, aiming to improve the accuracy of speech emotion recognition, enhance the real-time performance of speech recognition and its adaptability to different languages, dialects and accents.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] In a first aspect, the present application provides a speech emotion recognition method based on a spectrogram, comprising:

[0011] Get the audio signal spectrum and extract the Mel frequency cepstral coefficient features;

[0012] The sentiment classification model is used to perform sentiment recognition according to the Mel-frequency cepstral coefficient characteristics; the sentiment classification model includes an input layer, a convolution layer, a pooling layer, a fully connected layer, a Dropout layer and an output layer in sequence.

[0013] According to an embodiment of the present invention, extracting Mel-frequency cepstral coefficient features includes:

[0014] Use the librosa.load function of the Librosa library to read the audio file and obtain the audio signal spectrum;

[0015] Call librosa.feature.mfcc function to extract 13-dimensional MFCC features from the audio signal spectrogram.

[0016] According to an embodiment of the present invention, the Mel-frequency cepstral coefficient feature is standardized, including:

[0017] Use np.mean to get the average value of MFCC features.

[0018] This process reduces the data dimension while retaining sentiment-related information and is applicable to multilingual environments.

[0019] According to an embodiment of the present invention, the convolutional layer includes three layers:

[0020] The first layer consists of 32 filters of size (3,3);

[0021] The second layer consists of 64 filters of size (3,3);

[0022] The third layer consists of 64 filters of size (3,3).

[0023] Compared with traditional CNN, the number and size of filters in the convolutional layer in this application are designed according to audio features.

[0024] According to an embodiment of the present invention, the pooling layer has two layers and the window size is (2, 2).

[0025] Pooling layers help reduce the number of parameters and computation while retaining important information of features.

[0026] According to an embodiment of the present invention, the fully connected layer has 64 neurons and uses a ReLU activation function. The fully connected layer is used to make decisions from the features extracted by the convolutional layer.

[0027] According to an embodiment of the present invention, the output layer is a single neuron, and a Sigmoid activation function is used.

[0028] According to an embodiment of the present invention, the identification result is complaint or non-complaint.

[0029] According to an embodiment of the present invention, the sentiment classification model uses a Hyperband algorithm to perform hyperparameter optimization, and the hyperparameters include the number of filters of the convolutional layer, regularization strength, and learning rate.

[0030] In a second aspect, the present invention discloses a speech emotion recognition system based on a spectrogram, the system applying any of the above-mentioned speech emotion recognition methods based on a spectrogram, comprising:

[0031] A feature extraction module is used to obtain the audio signal spectrum and extract Mel frequency cepstral coefficient features;

[0032] The speech recognition module is used to perform emotion recognition according to the Mel-frequency cepstral coefficient characteristics using an emotion classification model; the emotion classification model includes an input layer, a convolution layer, a pooling layer, a fully connected layer, a Dropout layer and an output layer in sequence.

[0033] It can be seen from the above technical solutions that the present invention discloses a method and system for speech emotion recognition based on spectrogram, which analyzes the spectrogram of speech signals through deep learning technology to achieve more accurate and real-time emotion recognition. Compared with the prior art, the beneficial effects include:

[0034] 1. Improve the recognition accuracy of multiple languages ​​and dialects: Spectrograms can capture the audio features of speech without being affected by language differences. Spectrogram analysis can reduce dependence on language and dialects and improve recognition accuracy. This can more accurately capture users' emotional changes and demands, thereby improving service quality.

[0035] 2. Enhanced adaptability to accents and pronunciation differences: Spectrum graph analysis technology can better adapt to different accents and pronunciation differences. Based on spectrum graph analysis, it can improve the adaptability and flexibility of the system.

[0036] 3. Improve system processing speed: By directly analyzing the spectrum graph, the speech-to-text step is omitted, which can effectively improve the real-time processing capability to meet large-scale, high-concurrency processing needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 A flow chart of the speech emotion recognition method based on spectrogram provided by the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The embodiment of the present invention discloses a speech emotion recognition method and system based on a spectrogram, which aims to identify the emotional state by directly analyzing the spectrogram of the speech, so as to solve the problems of insufficient multi-language support, poor accent adaptability, slow processing speed and weak emotion recognition ability in the prior art.

[0041] Embodiment 1

[0042] This application is based on the speech emotion recognition method of the spectrogram, such as Figure 1 , the steps include:

[0043] Get the audio signal spectrum and extract the Mel frequency cepstral coefficient features;

[0044] The sentiment classification model is used to perform sentiment recognition according to the Mel-frequency cepstral coefficient characteristics; the sentiment classification model includes an input layer, a convolution layer, a pooling layer, a fully connected layer, a Dropout layer and an output layer in sequence.

[0045] The core of the present invention is to use the MFCC (Mel-Frequency Cepstral Coefficient) features in audio files for emotion recognition without relying on the speech-to-text process.

[0046] In this embodiment, the MFCC features in the audio file are automatically extracted through the Librosa library. These features can capture the audio features of the speech and are used as input data of the neural network after being standardized.

[0047] The extraction steps are as follows:

[0048] Use the librosa.load function of the Librosa library to read the audio file and obtain the audio signal spectrum, that is, the audio signal y and the sampling rate sr;

[0049] Call librosa.feature.mfcc function to extract 13-dimensional MFCC features from the audio signal spectrogram.

[0050] In order to optimize the above technical solution, the Mel frequency cepstral coefficient features are standardized, including:

[0051] The average value of the extracted MFCC features is calculated, and np.mean is used to obtain the average value of the MFCC features, which is used as the input feature for sentiment classification.

[0052] This process retains emotion-related information while reducing the data dimension and is suitable for multilingual environments because the extraction of MFCC features does not depend on the specific content of the language.

[0053] Furthermore, in this embodiment, the emotion classification model uses a specific convolutional neural network (CNN) architecture that is optimized for audio emotion recognition tasks. The design of the model takes into account the characteristics of audio data and places special emphasis on its application in multilingual and dialect environments.

[0054] In a specific embodiment, the model structure includes:

[0055] Input Layer:

[0056] Parameters: Accepts preprocessed audio features, specifically the average value of MFCC features, with a shape of (13, 1, 1).

[0057] Different from traditional image processing CNNs, the input layer directly processes audio features rather than raw audio signals or image data.

[0058] Convolutional Layer:

[0059] First layer: 32 filters of size (3,3), using ReLU activation function;

[0060] Second layer: 64 filters of size (3,3), using ReLU activation function;

[0061] The third layer: 64 filters of size (3,3) using the ReLU activation function.

[0062] Compared with traditional CNN, the convolutional layer in this application is specially designed for audio features, and the number and size of filters are selected based on the requirements of the audio emotion recognition task.

[0063] Pooling layer:

[0064] Including the first and second layers, the maximum pooling window size is (2,2);

[0065] Pooling layers help reduce the number of parameters and computation while maintaining important information of features, which is especially important for processing high-dimensional audio data.

[0066] Fully connected layer:

[0067] The number of neurons is 64 and the ReLU activation function is used.

[0068] In this application, the fully connected layer is used to make decisions from the features extracted by the convolutional layer, and the number of neurons is selected based on experiments and performance tests.

[0069] Dropout layer:

[0070] In this embodiment, the Dropout rate is 0.5.

[0071] Dropout layers help reduce overfitting, especially when dealing with complex audio emotion recognition tasks.

[0072] Output layer:

[0073] In this embodiment, the output layer is a single neuron, and a Sigmoid activation function is used to output the sentiment classification result.

[0074] The design of the output layer in this application is simple and effective, and is suitable for binary classification problems, that is, judging whether the emotional state belongs to the "complaint" category.

[0075] In one embodiment,

[0076] The sentiment classification model of this application uses the Hyperband algorithm to optimize hyperparameters, and the hyperparameters include the number of filters in the convolutional layer, regularization strength, and learning rate.

[0077] Compared with the traditional manual hyperparameter tuning, the Hyperband algorithm provides a more efficient and systematic hyperparameter optimization method, which can find the best hyperparameter combination and improve the performance and generalization ability of the model.

[0078] In the real-time emotion recognition process, the audio stream is first obtained, the MFCC features of the audio block are extracted, and then input into the lightweight emotion classification model. Based on the emotion classification results output by the model, the user's emotional state is displayed in real time without the need for speech-to-text conversion. This process is suitable for multilingual environments because the model directly extracts emotional features from the spectrogram and is not affected by language differences.

[0079] Spectrum graph analysis technology can better adapt to different accents and pronunciation differences. The present invention can significantly improve the accuracy, adaptability and flexibility of emotion recognition by directly analyzing the spectrum graph.

[0080] At the same time, by directly analyzing the spectrogram, the transcription steps are reduced and the real-time processing capability is improved. In addition, this application only extracts sound features to determine the emotional state, and does not involve the content of the call, which can well protect user privacy.

[0081] Embodiment 2

[0082] The present invention discloses a speech emotion recognition system based on a spectrogram, which applies any of the above-mentioned speech emotion recognition methods based on a spectrogram, and comprises:

[0083] A feature extraction module is used to obtain the audio signal spectrum and extract Mel frequency cepstral coefficient features;

[0084] The speech recognition module is used to perform emotion recognition according to the Mel-frequency cepstral coefficient characteristics using an emotion classification model; the emotion classification model includes an input layer, a convolution layer, a pooling layer, a fully connected layer, a Dropout layer and an output layer in sequence.

[0085] Preferably, the device further comprises an audio acquisition module for acquiring or recording audio files.

[0086] In one embodiment, the real-time audio emotion recognition system of the present invention is deployed in the customer service system of a power grid company to identify the emotional state by analyzing the customer's voice spectrogram to determine whether the customer is likely to make a complaint.

[0087] Scenario description: The customer service department of a power grid company often receives calls from customers, some of whom may complain due to service issues or other reasons. In order to improve customer satisfaction and service quality, the company needs a technology that can identify the customer's emotional state in real time so that the customer service team can intervene in time to deal with potential problems.

[0088] The implementation process includes:

[0089] 1. System deployment: The real-time audio emotion recognition model of the present invention is deployed in the customer service system of the power grid company. The model identifies the emotional state by analyzing the customer's voice spectrogram and determines whether the customer is likely to make a complaint.

[0090] 2. Audio collection: When a customer calls the service hotline, the system collects the customer's voice data in real time through the microphone.

[0091] 3. Feature extraction: The system uses spectrogram analysis technology to extract emotional features from real-time audio signals. These features are processed and used as input data for the neural network.

[0092] 4. Emotion classification: The system uses a convolutional neural network (CNN) to classify the spectrogram features to determine whether the customer's emotional state belongs to the "complaint" emotion. The model can identify complaint emotions such as anger, sadness, fear, and disgust, as well as non-complaint emotions such as happiness, neutrality, surprise, and naturalness.

[0093] 5. Real-time feedback: Based on the sentiment classification results output by the model, the system displays the customer’s emotional state in real time. If a complaint sentiment is detected, the system will immediately notify the customer service team so that they can respond quickly and take effective measures to resolve the problem.

[0094] 6. Customer service intervention: The customer service team prioritizes potential complaint calls based on the instant feedback provided by the system. This enables the customer service team to optimize resource allocation and improve work efficiency.

[0095] 7. Effect evaluation: By implementing the technology of the present invention, the power grid company can significantly improve the accuracy of complaint identification and real-time processing capabilities. This not only improves customer satisfaction and loyalty, but also reduces operating costs and enhances the company's brand image and market competitiveness.

[0096] When the present invention is deployed to the customer service system, it can quickly solve problems and improve customer satisfaction by timely identifying and responding to customer dissatisfaction; at the same time, it enables the customer service team to give priority to handling potential complaint calls based on system feedback, optimize resource allocation, and improve work efficiency.

[0097] Furthermore, automated emotion recognition reduces reliance on manual monitoring and analysis, lowers labor costs, and does not involve call content, thus fully protecting user privacy.

[0098] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0099] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech emotion recognition method based on spectrogram, characterized in that: Get the audio signal spectrum and extract the Mel frequency cepstral coefficient features; The sentiment classification model is used to perform sentiment recognition according to the Mel-frequency cepstral coefficient characteristics; the sentiment classification model includes an input layer, a convolution layer, a pooling layer, a fully connected layer, a Dropout layer and an output layer in sequence.

2. The speech emotion recognition method according to claim 1, characterized in that: Extract Mel frequency cepstral coefficient features, including: Use the librosa.load function of the Librosa library to read the audio file and obtain the audio signal spectrum; Call librosa.feature.mfcc function to extract 13-dimensional MFCC features from the audio signal spectrogram.

3. The speech emotion recognition method according to claim 2, characterized in that: Mel frequency cepstral coefficient features are standardized, including: Use np.mean to get the average value of MFCC features.

4. The speech emotion recognition method according to claim 1, characterized in that: The convolutional layer consists of three layers: The first layer consists of 32 filters of size (3,3); The second layer consists of 64 filters of size (3,3); The third layer consists of 64 filters of size (3,3).

5. The speech emotion recognition method according to claim 1, characterized in that: The pooling layer has two layers and the window size is (2,2).

6. The speech emotion recognition method according to claim 1, characterized in that: The fully connected layer has 64 neurons and uses the ReLU activation function.

7. The speech emotion recognition method according to claim 1, characterized in that: The output layer is a single neuron, using the Sigmoid activation function.

8. The speech emotion recognition method according to claim 1, characterized in that: The identification results are complaint and non-complaint.

9. The speech emotion recognition method according to claim 1, characterized in that: The sentiment classification model uses the Hyperband algorithm to optimize hyperparameters, and the hyperparameters include the number of filters of the convolutional layer, the regularization strength, and the learning rate.

10. A speech emotion recognition system based on spectrogram, characterized in that: The method for speech emotion recognition based on a spectrogram according to any one of claims 1 to 9 is applied, comprising: A feature extraction module is used to obtain the audio signal spectrum and extract Mel frequency cepstral coefficient features; The speech recognition module is used to perform emotion recognition according to the Mel-frequency cepstral coefficient characteristics using an emotion classification model; the emotion classification model includes an input layer, a convolution layer, a pooling layer, a fully connected layer, a Dropout layer and an output layer in sequence.

Images