Voice emotion recognition method, device, equipment and medium

Abstract

The application provides a speech emotion recognition method, device and equipment and medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining a speech signal from a user; converting the speech signal into time series data and extracting multiple dimensions of acoustic features from the speech signal, wherein the acoustic features are used to reflect the physical properties of the speech signal; performing feature extraction on the time series data and the multiple dimensions of acoustic features by using a pre-trained convolutional neural network (CNN) model to obtain target features; and determining an emotion type recognition result according to the target features by using a pre-trained recurrent neural network (RNN) model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for voice emotion recognition. Background Technology

[0002] With the rapid development of artificial intelligence technology, voice emotion recognition has shown enormous application potential in various fields such as human-computer interaction, intelligent customer service, and psychological counseling. Taking intelligent customer service as an example, by recognizing the user's emotional state, the intelligent customer service system can provide more considerate and personalized services, thereby improving the user experience.

[0003] However, traditional speech emotion recognition methods mainly rely on manually designed feature extraction (i.e., hand-designed features) and classification algorithms, which suffer from incomplete feature extraction and low recognition accuracy. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a voice emotion recognition method, apparatus, device, and medium to overcome or at least partially solve the above problems.

[0005] A first aspect of this application provides a voice emotion recognition method, the method comprising:

[0006] Acquire voice signals from users;

[0007] The speech signal is converted into time series data, and multiple acoustic features are extracted from the speech signal. These acoustic features are used to reflect the physical properties of the speech signal.

[0008] The target features are obtained by extracting features from the time series data and the acoustic features of the multiple dimensions using a pre-trained convolutional neural network (CNN) model.

[0009] The emotion type is determined based on the target features using a pre-trained recurrent neural network (RNN) model.

[0010] As one possible implementation, the method further includes:

[0011] Acquire multiple speech signal samples;

[0012] For each of the speech signal samples, the language expression of the speech signal sample is analyzed to determine the first match between the language expression and each preset emotion type, which includes: joy, sadness, anger, surprise, fear, calmness, disgust, love, anxiety, embarrassment, confusion, contempt, longing, comfort, and expectation.

[0013] For each of the speech signal samples, the sound attributes of the speech signal samples are analyzed to determine a second matching situation between the sound attributes and each preset emotion type;

[0014] Based on the first and second matching results corresponding to each of the plurality of speech signal samples, the emotion type label corresponding to each of the plurality of speech signal samples is determined from each of the preset emotion types;

[0015] The CNN model and the RNN model are trained using the multiple speech signal samples and their corresponding emotion type labels.

[0016] As one possible implementation, the step of analyzing the language expression of the speech signal sample and determining the first matching situation between the language expression and each preset emotion type includes:

[0017] Based on the frequency of occurrence of the words associated with each preset emotion type in the language expression, a first matching situation between the language expression and each preset emotion type is determined.

[0018] As one possible implementation, the step of analyzing the sound attributes of the speech signal sample and determining a second matching condition between the sound attributes and each preset emotion type includes:

[0019] The attribute features of the voice attributes associated with each preset emotion type are matched with the attribute features of the voice attributes of the speech signal sample to obtain a second matching result between the voice attributes and each preset emotion type. The voice attributes include: tone and speech rate.

[0020] As one possible implementation, the CNN model includes:

[0021] The input layer is used to receive the time-series data and the feature map composed of the acoustic features of the multiple dimensions;

[0022] A convolutional unit is used to extract features from the feature map layer by layer, from shallow features to deep features;

[0023] One or more fully connected layers are used to process the features extracted by the convolutional units to obtain the target features;

[0024] The output layer is used to output the target features.

[0025] As one possible implementation, the convolutional unit includes multiple convolutional layers connected by activation function layers and pooling layers.

[0026] As one possible implementation, converting the speech signal into time-series data and extracting multi-dimensional acoustic features from the speech signal includes:

[0027] The speech signal is preprocessed and then converted into time series data. The preprocessing includes: noise reduction, audio format standardization, and sampling rate unification.

[0028] Multiple acoustic features are extracted from the preprocessed speech signal. These multiple acoustic features include: spectral features, Mel-frequency cepstral coefficients, and spectrograms.

[0029] A second aspect of this application provides a voice emotion recognition device, the device comprising:

[0030] The signal acquisition module is used to acquire voice signals from the user;

[0031] The signal processing module is used to convert the speech signal into time series data and extract multiple dimensions of acoustic features from the speech signal, the acoustic features being used to reflect the physical properties of the speech signal;

[0032] The feature extraction module is used to extract features from the time series data and the acoustic features of the multiple dimensions using a pre-trained convolutional neural network (CNN) model to obtain target features.

[0033] The emotion recognition module is used to determine the emotion type recognition result based on the target features using a pre-trained recurrent neural network (RNN) model.

[0034] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the voice emotion recognition method described in the first aspect.

[0035] A fourth aspect of this application provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the voice emotion recognition method as described in the first aspect.

[0036] The embodiments of this application have the following advantages: by introducing a CNN model, deeper features (i.e., target features) are further extracted from the time series data and the acoustic features of multiple dimensions, which can avoid the subjectivity and incompleteness of manually designed feature extraction in traditional methods; furthermore, by introducing an RNN model, the recognition result of the emotion type is automatically determined based on the target features extracted by the CNN model, thereby realizing speech emotion recognition in a hybrid model manner, which can avoid the limitations of using a single model, and improve the robustness of emotion classification by leveraging the adaptability and generalization ability of the model; thus, the accuracy of speech emotion recognition can be effectively improved. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the implementation of a speech emotion recognition method in this application.

[0039] Figure 2 This is a schematic diagram of the architecture of a voice emotion recognition system according to an embodiment of this application;

[0040] Figure 3 This is a schematic diagram illustrating the implementation process of a speech emotion recognition method in an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of the structure of a deep learning model in an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of the structure of a voice emotion recognition device according to an embodiment of this application;

[0043] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0044] To facilitate understanding of the technical solutions provided in this application, the main technical concepts involved in the embodiments of this application are briefly described below.

[0045] Convolutional Neural Network (CNN): A type of neural network specifically designed to process data with a grid-like structure, such as images. CNNs mimic the human brain's visual processing mechanisms, using convolutional operations to extract features from the data, thereby enabling tasks such as classification and recognition of images and other data.

[0046] Recurrent Neural Network (RNN): A type of neural network specifically designed for processing sequential data. Unlike traditional neural networks (such as fully connected neural networks or convolutional neural networks), RNNs can remember previous information and use this information to influence the output of subsequent nodes. This characteristic makes RNNs perform exceptionally well when processing time-series data (such as text, video sequences, etc.).

[0047] Handcrafted features are a technique commonly used in fields such as machine learning, computer vision, and signal processing. They refer to the extraction of feature representations that are useful for a specific task from data through manual analysis and selection.

[0048] With the rapid development of artificial intelligence technology, voice emotion recognition has shown great application potential in various fields such as human-computer interaction, intelligent customer service, and psychological counseling. Traditional voice emotion recognition methods mainly rely on manually designed feature extraction and classification algorithms, which results in at least the following shortcomings:

[0049] ① Insufficient accuracy: Traditional speech emotion recognition methods often fail to capture all the subtle emotional changes in speech signals; this is mainly because they rely on limited hand-designed features, which are difficult to fully reflect the emotional information in speech.

[0050] ② Weak generalization ability: Traditional speech emotion recognition methods have weak generalization ability, meaning they may perform poorly in new environments or on new datasets. This is mainly because the feature extraction and classification algorithm design of traditional methods often depend on specific datasets and environmental conditions. When faced with new and unseen speech samples, their recognition performance may drop significantly.

[0051] ③ Low efficiency: Traditional speech emotion recognition methods may have low processing efficiency. This is mainly because manually designed feature extraction and classification algorithms may involve complex preprocessing steps and feature extraction processes, as well as requiring significant computational resources for emotion classification to ensure the accuracy of the results. These steps may increase system latency and power consumption, thus limiting its performance in real-time applications.

[0052] ④ Poor flexibility: Traditional voice emotion recognition methods are not very flexible, meaning they may be difficult to adapt to different application scenarios and changing needs. This is mainly because they are often designed to be optimized for specific tasks and lack sufficient flexibility and scalability to cope with new application scenarios or changing needs.

[0053] ⑤ High data requirements: Traditional speech emotion recognition methods require a large amount of labeled data for algorithm design (or parameter determination of a single algorithm model) to ensure its recognition accuracy. However, in practical applications, obtaining high-quality labeled data is often a challenging task, resulting in insufficient or biased training data, which affects the accuracy and generalization ability of emotion recognition.

[0054] To address the problems existing in the aforementioned related technologies, this application proposes a speech emotion recognition method, apparatus, device, and medium. By introducing a hybrid model composed of CNN and RNN models for speech emotion recognition, it can solve the problems of insufficient accuracy, weak generalization ability, low efficiency, poor flexibility, and large data requirements of traditional speech emotion recognition methods based on manually designed feature extraction and classification algorithms. This enables speech emotion recognition to be better applied in multiple fields such as human-computer interaction, intelligent customer service, and psychological counseling, thereby improving user experience.

[0055] The following description, in conjunction with the accompanying drawings, details a speech emotion recognition method, apparatus, device, and medium provided in this application through some embodiments and application scenarios.

[0056] Firstly, referring to Figure 1 The diagram shown is an implementation flowchart of a speech emotion recognition method provided in this application embodiment. The method includes the following steps:

[0057] Step S11: Obtain the voice signal from the user.

[0058] In practice, user-input voice signals can be obtained through interfaces such as web pages, for example, voice signals from users asking questions or providing feedback to the intelligent customer service system.

[0059] Step S12: Convert the speech signal into time series data and extract multiple dimensions of acoustic features from the speech signal. The acoustic features are used to reflect the physical properties of the speech signal.

[0060] In practice, the acquired speech signals are arranged in chronological order (and at certain time intervals) to obtain time-series data suitable for processing by deep learning models (such as CNN and RNN models). Multiple acoustic features are extracted from the speech signals, such as fundamental frequency, short-time energy, formants, and Mel-Frequency Cepstral Coefficients (MFCCs), which reflect the physical properties of the speech signals. This allows the RNN model to combine the time-series data and the multiple acoustic features to more accurately and comprehensively extract deep-level features related to emotion (i.e., the target features).

[0061] Step S13: Using a pre-trained convolutional neural network (CNN) model, feature extraction is performed on the time series data and the acoustic features of the multiple dimensions to obtain the target features.

[0062] In practice, the target features extracted by the CNN model are input into the RNN model for sequence modeling and classification to identify the emotion type reflected in the speech signal. Thus, speech emotion recognition is achieved based on a hybrid model composed of the CNN and RNN models. Understandably, this hybrid model can achieve better recognition performance than a single model.

[0063] Step S14: Determine the recognition result of the emotion type based on the target features using a pre-trained recurrent neural network (RNN) model.

[0064] In practice, the results of emotion type recognition can be output to the user in the form of text or voice; alternatively, strategy decisions and adjustments can be made based on the results of emotion type recognition. For example, the dialogue strategy of the intelligent customer service system can be adjusted based on the results of emotion type recognition so that the intelligent customer service system can provide more considerate and personalized services, thereby improving the user experience.

[0065] The technical solution of this application, by introducing a CNN model, further extracts deep features (i.e., target features) from the time series data and the acoustic features of multiple dimensions, which can avoid the subjectivity and incompleteness of manually designed feature extraction in traditional methods. Furthermore, by introducing an RNN model, the recognition result of the emotion type is automatically determined based on the target features extracted by the CNN model. Thus, speech emotion recognition is achieved in a hybrid model manner, which can avoid the limitations of using a single model, and the robustness of emotion classification can be improved by leveraging the adaptability and generalization ability of the model. In this way, the accuracy of speech emotion recognition can be effectively improved.

[0066] As one possible implementation, converting the speech signal into time-series data and extracting multi-dimensional acoustic features from the speech signal includes:

[0067] Step S121: Preprocess the speech signal and convert the preprocessed speech signal into time series data. The preprocessing includes: noise reduction, audio format standardization and sampling rate unification.

[0068] In practical implementation, denoising speech signals first requires estimating the characteristics of the background noise. For example, noise estimation can be achieved by analyzing noise in inactive segments of the speech signal (such as silent segments). Then, based on the noise estimation results, an appropriate algorithm is used to suppress the noise. Thus, by removing noise components from the speech signal, the clarity and recognizability of the speech can be improved.

[0069] Taking spectral subtraction as an example, based on the separability of noise signals and speech signals in the frequency domain, the noise spectrum is subtracted from the spectrum of the noisy speech signal to obtain a relatively clean speech spectrum, thereby achieving denoising of the speech signal.

[0070] For audio format standardization of speech signals, a unified audio format (such as WAV) is first determined based on the requirements of subsequent processing. Then, audio processing software or libraries (such as SOX) are used to convert the audio files used to store the speech signals into the unified audio format. Thus, by converting the acquired audio files of different formats into a unified format, subsequent speech signal processing and analysis are facilitated.

[0071] To unify the sampling rate of speech signals, a suitable sampling rate (i.e., the target sampling rate, such as 44.1kHz) is first determined based on the requirements of subsequent processing. Then, resampling techniques are used to convert the current sampling rate of the speech signal to the target sampling rate. During the sampling rate conversion process, interpolation or decimation operations can be introduced to ensure the continuity and smoothness of the speech signal. Thus, by converting speech signals with different sampling rates into a unified sampling rate, the requirements of subsequent processing algorithms are met.

[0072] Step S122: Extract multiple dimensions of acoustic features from the preprocessed speech signal, including: spectral features, Mel frequency cepstral coefficients, and spectrogram.

[0073] Mel frequency cepstral coefficients are a widely used speech feature that can simulate the human ear's perception of frequency; spectrograms are used to show how the frequency of a speech signal changes over time, which helps to capture dynamic features; the spectral features can be obtained by converting the speech signal from the time domain to the frequency domain based on the Short-Time Fourier Transform (STFT).

[0074] As one possible implementation, the method further includes:

[0075] Step S21: Acquire multiple speech signal samples.

[0076] In specific implementation, refer to Figure 2 The schematic diagram of the speech emotion recognition system shown can be used to execute the speech emotion recognition method provided in the embodiments of this application.

[0077] Specifically, the data acquisition and preprocessing layer of the speech emotion recognition system first acquires open-source and public speech data from the internet to obtain multiple speech signal samples. These multiple speech signal samples should cover the various preset emotion types to ensure the diversity and balance of the samples. Then, preprocessing operations such as noise reduction, audio format standardization, and uniform sampling rate are performed on the acquired multiple speech signal samples to improve data quality and consistency.

[0078] Step S22: For each of the speech signal samples, analyze the language expression of the speech signal sample and determine the first matching situation between the language expression and each preset emotion type, wherein the preset emotion types include: joy, sadness, anger, surprise, fear, calmness, disgust, love, anxiety, embarrassment, confusion, contempt, longing, comfort and expectation.

[0079] In practice, the degree of matching (such as matching score) between the language expression in the speech signal sample and each preset emotion type can be determined based on experience, thereby obtaining the first matching result.

[0080] Alternatively, the first matching situation between the language expression and each preset emotion type can be determined based on the frequency of occurrence of the words associated with each preset emotion type in the language expression. Specifically:

[0081] The more words associated with joy that are included (i.e. appear) in the language expression, the higher the degree of matching (e.g., matching score) between the language expression and the preset emotion type of joy. Words associated with joy can be positive and bright expressions such as "That's great!" or "I'm so happy!"

[0082] The more words associated with sadness the language expression contains, the higher the degree of match (e.g., match score) between the language expression and the preset emotional type of sadness. Words associated with sadness can include expressions of loss, regret, or pain, such as "I'm so sad" or "This is so heartbreaking."

[0083] The more words associated with anger contained in the language expression, the higher the degree of match (e.g., match score) between the language expression and the preset emotion type of anger. Words associated with anger can include phrases like "How could you do this!" or "I'm angry!" that express emotions of blame, dissatisfaction, or threat.

[0084] The more words associated with surprise that are included in the language expression, the higher the degree of match (e.g., match score) between the language expression and the preset emotional type of surprise. Words associated with surprise can include expressions of unexpectedness such as "Wow!" or "Really?".

[0085] The more fear-related words included in the language expression, the higher the degree of match (e.g., match score) between the language expression and the preset emotional type of fear. These fear-related words can include phrases such as "I'm scared" or "It's so scary."

[0086] The more words associated with calmness contained in the language expression, the higher the degree of match (e.g., match score) between the language expression and the preset emotional type of calmness. Words associated with calmness can be those without strong emotional connotations (i.e., expressing a peaceful and stable emotion).

[0087] The more words associated with disgust that are included in the language expression, the higher the degree of matching (e.g., matching score) between the language expression and the preset emotional type of disgust. The words associated with disgust can be words expressing dissatisfaction or aversion, such as "disgusting" or "unbearable."

[0088] The more words associated with love contained in the language expression, the higher the degree of matching (e.g., matching score) between the language expression and the preset emotional type of love. Words associated with love can include phrases like "I love you" and "You're great," which express emotions of tenderness, care, and praise.

[0089] The more anxiety-related words contained in the language expression, the higher the degree of match (e.g., match score) between the language expression and the preset emotional type of anxiety. These anxiety-related words can include expressions such as "I'm nervous" or "I'm worried something will go wrong," which convey feelings of worry, unease, or tension.

[0090] The more words associated with embarrassment contained in the language expression, the higher the degree of matching (e.g., matching score) between the language expression and the preset emotional type of embarrassment. The words associated with embarrassment can be words used for apology, denial, or changing the subject (i.e., to conceal embarrassment), such as "excuse me" or "that's not what I meant."

[0091] The more words associated with confusion contained in the language expression, the higher the degree of matching (e.g., matching score) between the language expression and the preset emotional type of confusion. The words associated with confusion can be phrases like "What does this mean?" or "I don't quite understand," used to ask questions, repeat, or express confusion.

[0092] The more words associated with contempt contained in the language expression, the higher the degree of match (e.g., match score) between the language expression and the preset emotional type of contempt. Words associated with contempt can include phrases like "That's ridiculous" or "You're such an idiot," which express feelings of belittling, mockery, or disdain.

[0093] The more words associated with desire contained in the language expression, the higher the degree of match (e.g., match score) between the language expression and the preset emotional type of desire. Words associated with desire can be expressions such as "I really want that," or "Can you give it to me?" that express expectation, longing, or request.

[0094] The more words associated with comfort contained in the language expression, the higher the degree of match (e.g., match score) between the language expression and the preset emotional type of comfort. Words associated with comfort can include expressions such as "it's finally over," "thank you for helping me," etc., that express feelings of relief, reassurance, or gratitude.

[0095] The more words associated with expectation contained in the language expression, the higher the degree of matching (e.g., matching score) between the language expression and the preset emotional type of expectation. The words associated with expectation can be those expressing emotions such as anticipation, excitement, or curiosity about future events, such as "I'm really looking forward to tomorrow" or "What do you think will happen?"

[0096] For each speech signal sample, the degree of matching (e.g., matching score) between the determined language expression of the speech signal sample and each preset emotion type is summarized to obtain the first matching situation corresponding to the speech signal sample.

[0097] Step S23: For each of the speech signal samples, analyze the sound attributes of the speech signal samples to determine the second matching situation between the sound attributes and each preset emotion type.

[0098] In practice, the sound attributes of the speech signal samples can be analyzed based on experience (e.g., the emotional tension of the voice, the tendency of the expressed emotion, etc.) to determine the matching score between the sound attributes and each preset emotion type, thereby obtaining the second matching result.

[0099] Alternatively, the attribute features of the voice attributes associated with each of the preset emotion types can be matched with the attribute features of the voice attributes of the speech signal sample to obtain a second matching result between the voice attributes and each preset emotion type. The voice attributes include at least: tone and speech rate, specifically:

[0100] The higher the degree of match between the voice attributes associated with joy (such as rising intonation and slightly faster speech rate, as well as vitality and enthusiasm in the voice) and the voice attributes of the speech signal sample, the higher the degree of match (e.g., matching score) between the voice attributes of the speech signal sample and the preset emotion type of joy.

[0101] The higher the degree of match between the voice attributes associated with sadness (such as low tone and slow speech rate, as well as trembling or sobbing) and the voice attributes of the speech signal sample, the higher the degree of match (e.g., match score) between the voice attributes of the speech signal sample and the preset emotion type of sadness.

[0102] The higher the degree of match between the voice attributes associated with anger, such as high-pitched tone and fast and rapid speech (and strong emotional tension in the voice), and the higher the degree of match (e.g., matching score) between the voice attributes of the speech signal sample and the preset emotion type of anger, the higher the degree of match (e.g., matching score) can be.

[0103] The higher the degree of match between the voice attributes associated with surprise, such as sudden rises or falls in tone and short pauses in speech rate, and the voice attributes of the speech signal sample, the higher the degree of match (e.g., match score) between the voice attributes of the speech signal sample and the preset emotion type of surprise.

[0104] The higher the degree of match between the voice attributes associated with fear, such as trembling tone and increased or intermittent speech (as well as a low and tense voice), and the voice attributes of the speech signal sample, the higher the degree of match (e.g., matching score) between the voice attributes of the speech signal sample and the preset emotion type of fear.

[0105] The higher the degree of matching between the voice attributes associated with calmness (such as a steady tone and moderate speaking speed, as well as a clear and gentle voice) and the voice attributes of the speech signal sample, the higher the degree of matching (e.g., matching score) between the voice attributes of the speech signal sample and the preset emotion type of calmness.

[0106] The characteristics of the voice attributes associated with disgust include a slightly sarcastic or dissatisfied tone and a speech rate that varies with emotion. The higher the degree of matching between the voice attributes of the speech signal sample and the characteristics of the voice attributes, the higher the degree of matching (e.g., matching score) between the voice attributes of the speech signal sample and the preset emotion type of disgust.

[0107] The higher the degree of matching between the voice attributes associated with love (such as a gentle tone and moderate speaking speed, and a voice full of warmth and emotion) and the voice attributes of the speech signal sample, the higher the degree of matching (e.g., matching score) between the voice attributes of the speech signal sample and the preset emotion type of love.

[0108] The characteristics of the voice attributes associated with anxiety, such as a slightly hurried tone and a slightly fast or unstable speaking speed (as well as a sense of tension in the voice), are considered to be higher if the voice attributes of the voice signal sample match the preset emotion type of anxiety (e.g., the matching score).

[0109] The higher the degree of match between the voice attributes associated with embarrassment (such as a soft or low tone of voice, and a hint of discomfort in the voice) and the voice attributes of the speech signal sample, the higher the degree of match (e.g., a matching score) between the voice attributes of the speech signal sample and the preset emotion type of embarrassment.

[0110] The higher the degree of match between the voice attributes associated with confusion (indicating questioning) and the slowing of speech rate (indicating seeking more information) and the voice attributes of the speech signal sample, the higher the degree of match (e.g., match score) between the voice attributes of the speech signal sample and the preset emotional type of confusion.

[0111] The characteristics of the voice attributes associated with contempt: a tone of voice with sarcasm or contempt and a steady speaking speed (as well as a tone of dissatisfaction, etc.). The higher the degree of matching between the voice attributes of the voice signal sample and the characteristics of the voice attributes, the higher the degree of matching (e.g., matching score) between the voice attributes of the voice signal sample and the preset emotion type of contempt.

[0112] The higher the degree of matching between the voice attributes associated with desire and the voice characteristics of the speech signal sample, such as a slightly urgent or expectant tone and a slightly faster speaking speed (to express inner desire), the higher the degree of matching (e.g., matching score) between the voice attributes of the speech signal sample and the preset emotional type of desire.

[0113] The higher the degree of match between the voice attributes associated with comfort and the characteristics of a soft and relaxed tone and a slowed speech rate (to express inner peace), the higher the degree of match (e.g., match score) between the voice attributes of the speech signal sample and the preset emotional type of comfort.

[0114] The higher the degree of match between the voice attributes associated with expectation and the voice characteristics of the speech signal sample, such as rising intonation with excitement and slightly faster speech rate (to express inner excitement), the higher the degree of match (e.g., matching score) between the voice attributes of the speech signal sample and the preset emotional type of expectation.

[0115] For each speech signal sample, the attribute features of the speech signal sample's sound attributes can be identified according to a pre-trained deep learning model. Then, the degree of matching between the speech signal sample's sound attributes and each preset emotion type can be determined (e.g., the matching score, which can be obtained by weighted sum of the matching scores corresponding to different attribute features). After summing up the obtained matching degrees, the second matching situation corresponding to the speech signal sample can be obtained.

[0116] Step S24: Based on the first matching situation and the second matching situation corresponding to each of the multiple speech signal samples, determine the emotion type label corresponding to each of the multiple speech signal samples from the multiple preset emotion types.

[0117] In practical implementation, the first and second matching situations corresponding to each of the multiple speech signal samples can be comprehensively considered by calculating weighted sums and taking intersections to obtain the preset emotion type with the highest matching degree for each of the multiple speech signal samples. Then, the preset emotion types with the highest matching degree are used as the emotion type labels corresponding to each of the multiple speech signal samples.

[0118] Step S25: Train the CNN model and the RNN model using the multiple speech signal samples and their corresponding emotion type labels.

[0119] In practical implementation, the acoustic feature extraction layer in the speech emotion recognition system first uses traditional speech signal processing methods to extract multiple dimensions of acoustic features from the multiple speech signal samples, such as fundamental frequency, short-time energy, formants and Mel frequency cepstral coefficients, and converts the multiple speech signal samples into a format suitable for deep learning model processing: time series data format.

[0120] The emotion recognition model layer in the system uses the acoustic features and time series obtained by processing the multiple speech signal samples by the acoustic feature extraction layer, as well as the emotion type labels corresponding to each of the multiple speech signal samples, to train the hybrid model composed of the CNN model and the RNN model. When the training conditions are met (such as reaching the number of iterations or the model performance meets the requirements), the training of the hybrid model ends, and the trained CNN model and RNN model are obtained.

[0121] Reference Figure 3 The diagram illustrates the implementation process of the speech emotion recognition method. In the subsequent model inference stage, the emotion recognition model layer extracts emotional features (i.e., target features) from the speech signal using a CNN model. The extracted emotional features are then transmitted to an RNN model for emotion type identification. The result output layer of the speech emotion recognition system then outputs the emotion type identification result in text or speech form to the user or intelligent customer service application system, allowing the user or application system to adjust its next strategy based on the identification result.

[0122] In this embodiment, a hybrid model consisting of the CNN model and the RNN model is trained using a labeled sentiment classification dataset. This allows the hybrid model to learn how to map input data to preset sentiment types. Compared to training the CNN model and the RNN model separately, this hybrid model-based training method can reduce the workload of data labeling. Furthermore, since hybrid models often achieve better recognition performance than single models, this hybrid model-based training method can also effectively reduce the amount of data required for model training convergence.

[0123] As one possible implementation, the CNN model includes:

[0124] The input layer is used to receive the time series data and the feature map composed of the acoustic features of the multiple dimensions, wherein different rows and columns of the feature map correspond to different time points and different data dimensions, respectively.

[0125] A convolutional unit is used to extract features from the feature map layer by layer, from shallow features to deep features;

[0126] One or more fully connected layers are used to process the features extracted by the convolutional units to obtain the target features;

[0127] The output layer is used to output the target features.

[0128] In specific implementation, the convolutional unit can stack multiple convolutional layers to achieve layer-by-layer feature extraction from shallow features to deep features, that is, to achieve layer-by-layer feature extraction from low-level features (extracted by shallow convolutional layers) to high-level features (which can be more complex speech patterns such as syllables and morphemes, extracted by deep convolutional layers).

[0129] One or more fully connected layers can be connected after the convolutional unit to further process (and classify) the high-level features extracted by the convolutional unit. For example, features extracted by different convolutional kernels in the convolutional unit can be fused to form a more comprehensive feature representation. This feature fusion helps to capture multiple characteristics in the speech signal, thereby improving the accuracy of subsequent recognition or classification.

[0130] Optionally, the convolutional unit includes multiple convolutional layers connected by activation function layers and pooling layers.

[0131] The convolutional layer captures local features in the speech signal through local connectivity and weight sharing mechanisms. Specifically, the convolutional layer slides a kernel across the input feature map, performing convolution operations with local regions to generate new feature maps. The pooling layer reduces the size of the feature map through downsampling while preserving important feature information. This helps reduce computational cost and improve the model's robustness. The pooling layer can be used to perform max pooling or average pooling operations.

[0132] In practice, the output of a convolutional layer is processed by an activation function layer and a pooling layer, respectively. The outputs of the activation function layer and the pooling layer are then processed by the next convolutional layer. In this way, multiple convolutional layers are connected through activation function layers and pooling layers to form a convolutional unit.

[0133] In this implementation, by introducing a combination of multiple convolutional layers, pooling layers, and activation function layers into the CNN model, it is ensured that the model can gradually extract higher-level features.

[0134] Optionally, the activation function layer may use nonlinear activation functions such as Rectified Linear Unit (ReLU) to increase the nonlinearity of the model, thereby helping to extract more complex features.

[0135] Taking the embedding of the above-mentioned voice emotion recognition system into an intelligent customer service system as an example, the implementation and application process of the voice emotion recognition method provided in this application embodiment will be illustrated.

[0136] When a user interacts with the customer service system via voice, the voice emotion recognition system first receives the user's input voice signal and performs preprocessing, acoustic feature extraction, and time-series data conversion. Specifically, after receiving the voice signal, the voice emotion recognition system first performs noise reduction to remove background noise and irrelevant background sounds, then removes silent segments to reduce computational load, and finally unifies the sampling rate to ensure data consistency, thus completing the preprocessing. Then, for the preprocessed voice signal, acoustic feature extraction methods such as MFCC are used to extract key features that reflect the emotional state of the voice (i.e., multi-dimensional acoustic features), and the voice signal is converted into time-series data.

[0137] Then, the speech emotion recognition system uses a CNN model to extract features from the feature map composed of time-series data and multiple dimensions of acoustic features. Next, an RNN model is used to identify the emotion type based on the features extracted by the CNN model. Specifically, the system inputs the feature map into a pre-trained deep learning model (i.e., a hybrid model composed of a CNN model and an RNN model). This model, through learning and analysis of multiple layers of neural networks, can identify the emotion type in the user's speech signal. For example, if the model analyzes that the user's tone is rapid and the pitch is high, it can obtain the emotion type "anger". The structural diagram of the deep learning model is shown below. Figure 4 As shown.

[0138] Finally, the intelligent customer service system outputs the recognition results determined by the voice emotion recognition system to customer service personnel in text form or automatically triggers corresponding service processes or adjusts response strategies. For example, if a user expresses anger, the intelligent customer service system can adopt a gentler and more patient response; if a user expresses happiness or satisfaction, the intelligent customer service system can provide a more positive and affirmative response. Furthermore, the voice emotion recognition system can compare the emotion type recognition results with the user's actual feedback to continuously optimize the parameters and performance of the deep learning model, improving the accuracy and robustness of voice emotion recognition. Experimental results show that this voice emotion recognition system exhibits excellent performance on various emotion recognition tasks, significantly improving the user experience of the intelligent customer service system.

[0139] It should be noted that the speech emotion recognition method provided in this application embodiment has at least the following application extensions:

[0140] 1. Enhanced Human-Computer Interaction Experience: In fields such as intelligent customer service, smart home, and virtual reality, accurately identifying users' emotional states (i.e., emotional types) can help provide users with more considerate and personalized services during human-computer interaction, thereby improving the user experience.

[0141] 2. Mental health support: In fields such as psychological counseling and psychotherapy, by accurately identifying the user's emotional state, it can help psychologists understand the patient's emotional changes, thereby providing strong support for clinical diagnosis and treatment.

[0142] 3. Lie detection and security testing: In the fields of law enforcement and security, by accurately identifying the user's emotional state, it is possible to help judge the psychological state of the test subject, thereby improving the accuracy and efficiency of lie detection and security testing.

[0143] Based on the above examples and embodiments, the speech emotion recognition method provided in this application can achieve at least the following optimization effects:

[0144] 1. Improve recognition accuracy: By introducing a CNN model to automatically extract deep features from speech signals, the subjectivity and incompleteness of manual feature extraction in traditional methods can be reduced, thereby improving the accuracy of speech emotion recognition.

[0145] 2. Enhanced generalization ability: Deep learning models such as CNN and RNN can learn more general feature representations, thereby enabling them to handle speech signals from different sources and of different qualities, thus improving the robustness of the system and making the recognition performance of the system more stable under different environments and different speaker conditions, with stronger generalization ability.

[0146] 3. Optimize algorithm structure: Using a hybrid model for training and inference can reduce the computational complexity of the model, improve the recognition speed, and reduce resource consumption. Furthermore, by combining model-related hardware acceleration technologies, it is possible to achieve real-time or near-real-time speech emotion recognition, meeting the needs of practical applications.

[0147] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of this application.

[0148] Secondly, Figure 5 This is a schematic diagram of the structure of a voice emotion recognition device according to an embodiment of this application. The device includes:

[0149] The signal acquisition module 510 is used to acquire voice signals from the user;

[0150] The signal processing module 520 is used to convert the speech signal into time series data and extract multiple dimensions of acoustic features from the speech signal, the acoustic features being used to reflect the physical properties of the speech signal.

[0151] The feature extraction module 530 is used to extract features from the time series data and the acoustic features of the multiple dimensions using a pre-trained convolutional neural network (CNN) model to obtain target features.

[0152] The emotion recognition module 540 is used to determine the recognition result of the emotion type based on the target features using a pre-trained recurrent neural network (RNN) model.

[0153] The technical solution of this application, by introducing a CNN model, further extracts deep features (i.e., target features) from the time series data and the acoustic features of multiple dimensions, which can avoid the subjectivity and incompleteness of manually designed feature extraction in traditional methods. Furthermore, by introducing an RNN model, the recognition result of the emotion type is automatically determined based on the target features extracted by the CNN model. Thus, speech emotion recognition is achieved in a hybrid model manner, which can avoid the limitations of using a single model, and the robustness of emotion classification can be improved by leveraging the adaptability and generalization ability of the model. In this way, the accuracy of speech emotion recognition can be effectively improved.

[0154] Optionally, the device further includes a model training module for performing the following steps:

[0155] Acquire multiple speech signal samples;

[0156] For each of the speech signal samples, the language expression of the speech signal sample is analyzed to determine the first match between the language expression and each preset emotion type, which includes: joy, sadness, anger, surprise, fear, calmness, disgust, love, anxiety, embarrassment, confusion, contempt, longing, comfort, and expectation.

[0157] For each of the speech signal samples, the sound attributes of the speech signal samples are analyzed to determine a second matching situation between the sound attributes and each preset emotion type;

[0158] Based on the first and second matching results corresponding to each of the plurality of speech signal samples, the emotion type label corresponding to each of the plurality of speech signal samples is determined from each of the preset emotion types;

[0159] The CNN model and the RNN model are trained using the multiple speech signal samples and their corresponding emotion type labels.

[0160] Optionally, the model training module is further configured to determine a first matching situation between the language expression and the preset emotion type based on the frequency of occurrence of the words associated with each preset emotion type in the language expression.

[0161] Optionally, the model training module is further configured to match the attribute features of the sound attributes associated with each preset emotion type with the attribute features of the sound attributes of the speech signal sample to obtain a second matching result between the sound attributes and each preset emotion type, wherein the sound attributes include: tone and speech rate.

[0162] Optionally, the CNN model includes:

[0163] The input layer is used to receive the time-series data and the feature map composed of the acoustic features of the multiple dimensions;

[0164] A convolutional unit is used to extract features from the feature map layer by layer, from shallow features to deep features;

[0165] One or more fully connected layers are used to process the features extracted by the convolutional units to obtain the target features;

[0166] The output layer is used to output the target features.

[0167] Optionally, the convolutional unit includes multiple convolutional layers connected by activation function layers and pooling layers.

[0168] Optionally, the signal processing module 520 is further configured to perform the following steps:

[0169] The speech signal is preprocessed and then converted into time series data. The preprocessing includes: noise reduction, audio format standardization, and sampling rate unification.

[0170] Multiple acoustic features are extracted from the preprocessed speech signal. These multiple acoustic features include: spectral features, Mel-frequency cepstral coefficients, and spectrograms.

[0171] It should be noted that the device embodiments are similar to the method embodiments, so the description is relatively simple. For relevant details, please refer to the method embodiments.

[0172] This application also provides an electronic device, see embodiments thereof. Figure 6 , Figure 6This is a schematic diagram of the electronic device proposed in an embodiment of this application. Figure 6 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus for communication. The memory 110 stores a computer program that can run on the processor 120 to implement the steps in the voice emotion recognition method disclosed in the embodiments of this application.

[0173] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the voice emotion recognition method disclosed in this application.

[0174] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the voice emotion recognition method disclosed in this application.

[0175] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0176] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0177] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, systems, devices, storage media, and program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0178] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0180] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0181] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0182] The above provides a detailed description of the speech emotion recognition method, apparatus, device, and medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A voice emotion recognition method, characterized in that, The method includes: Acquire voice signals from users; The speech signal is converted into time series data, and multiple acoustic features are extracted from the speech signal. These acoustic features are used to reflect the physical properties of the speech signal. The target features are obtained by extracting features from the time series data and the acoustic features of the multiple dimensions using a pre-trained convolutional neural network (CNN) model. The emotion type recognition result is determined based on the target features using a pre-trained recurrent neural network (RNN) model. Acquire multiple speech signal samples; For each of the speech signal samples, the language expression of the speech signal sample is analyzed to determine the first matching situation between the language expression and each preset emotion type. The first matching situation is the degree of matching between the language expression and each preset emotion type, which includes: joy, sadness, anger, surprise, fear, calmness, disgust, love, anxiety, embarrassment, confusion, contempt, longing, comfort, and expectation. For each of the speech signal samples, the sound attributes of the speech signal samples are analyzed to determine a second matching situation between the sound attributes and each of the preset emotion types. The second matching situation is the degree of matching between the sound attributes and each of the preset emotion types. The sound attributes include tone and speech rate. Based on the first and second matching results corresponding to each of the plurality of speech signal samples, the emotion type label corresponding to each of the plurality of speech signal samples is determined from each of the preset emotion types; The CNN model and the RNN model are trained using the multiple speech signal samples and their corresponding emotion type labels.

2. The method according to claim 1, characterized in that, The step of analyzing the language expression of the speech signal sample and determining the first match between the language expression and each preset emotion type includes: Based on the frequency of occurrence of the words associated with each preset emotion type in the language expression, a first matching situation between the language expression and each preset emotion type is determined.

3. The method according to claim 1, characterized in that, The step of analyzing the sound attributes of the speech signal samples and determining the second matching situation between the sound attributes and each preset emotion type includes: The attribute features of the voice attributes associated with each preset emotion type are matched with the attribute features of the voice attributes of the speech signal sample to obtain a second matching result between the voice attributes and each preset emotion type. The voice attributes include: tone and speech rate.

4. The method according to claim 1, characterized in that, The CNN model includes: The input layer is used to receive the time-series data and the feature map composed of the acoustic features of the multiple dimensions; A convolutional unit is used to extract features from the feature map layer by layer, from shallow features to deep features; One or more fully connected layers are used to process the features extracted by the convolutional units to obtain the target features; The output layer is used to output the target features.

5. The method according to claim 4, characterized in that, The convolutional unit includes multiple convolutional layers connected by activation function layers and pooling layers.

6. The method according to any one of claims 1-5, characterized in that, The process of converting the speech signal into time-series data and extracting multi-dimensional acoustic features from the speech signal includes: The speech signal is preprocessed and then converted into time series data. The preprocessing includes: noise reduction, audio format standardization, and sampling rate unification. Multiple acoustic features are extracted from the preprocessed speech signal. These multiple acoustic features include: spectral features, Mel-frequency cepstral coefficients, and spectrograms.

7. A voice emotion recognition device, characterized in that, The device includes: The signal acquisition module is used to acquire voice signals from the user; The signal processing module is used to convert the speech signal into time series data and extract multiple dimensions of acoustic features from the speech signal, the acoustic features being used to reflect the physical properties of the speech signal; The feature extraction module is used to extract features from the time series data and the acoustic features of the multiple dimensions using a pre-trained convolutional neural network (CNN) model to obtain target features. The emotion recognition module is used to determine the emotion type recognition result based on the target features using a pre-trained recurrent neural network (RNN) model. The model training module is used to perform the following steps: Acquire multiple speech signal samples; For each of the speech signal samples, the language expression of the speech signal sample is analyzed to determine the first matching situation between the language expression and each preset emotion type. The first matching situation is the degree of matching between the language expression and each preset emotion type, which includes: joy, sadness, anger, surprise, fear, calmness, disgust, love, anxiety, embarrassment, confusion, contempt, longing, comfort, and expectation. For each of the speech signal samples, the sound attributes of the speech signal samples are analyzed to determine a second matching situation between the sound attributes and each of the preset emotion types. The second matching situation is the degree of matching between the sound attributes and each of the preset emotion types. The sound attributes include tone and speech rate. Based on the first and second matching results corresponding to each of the plurality of speech signal samples, the emotion type label corresponding to each of the plurality of speech signal samples is determined from each of the preset emotion types; The CNN model and the RNN model are trained using the multiple speech signal samples and their corresponding emotion type labels.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the voice emotion recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the speech emotion recognition method as described in any one of claims 1 to 6.

Images