Speech emotion recognition method and device, equipment and medium

Abstract

The invention provides a voice emotion recognition method and device, equipment and a medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining a voice signal from a user; converting the voice signal into time sequence data, and extracting acoustic features of multiple dimensions from the voice signal, the acoustic features being used for reflecting physical attributes of the voice signal; through a pre-trained convolutional neural network CNN model, feature extraction is carried out on the time sequence data and the acoustic features of the multiple dimensions, and target features are obtained; and through a pre-trained recurrent neural network (RNN) model, determining a recognition result of the emotion type according to the target feature.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for speech emotion recognition. Background Art

[0002] With the rapid development of artificial intelligence technology, speech emotion recognition has shown great application potential in many fields such as human-computer interaction, intelligent customer service and psychological counseling. Taking intelligent customer service as an example, by identifying the user's emotional state, it is beneficial for the intelligent customer service system to provide more intimate and personalized services, thereby improving the user experience.

[0003] However, traditional speech emotion recognition methods mainly rely on artificially designed feature extraction (i.e., manually designed features) and classification algorithms, which have problems such as incomplete feature extraction and low recognition accuracy. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide a speech emotion recognition method, apparatus, device and medium to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, a method for speech emotion recognition is provided, the method comprising: Acquire a voice signal from a user; Converting the speech signal into time series data, and extracting acoustic features of multiple dimensions from the speech signal, wherein the acoustic features are used to reflect the physical properties of the speech signal; Extracting features of the time series data and the acoustic features of the multiple dimensions through a pre-trained convolutional neural network (CNN) model to obtain target features; The recognition result of the emotion type is determined according to the target feature through a pre-trained recurrent neural network (RNN) model.

[0006] As a possible implementation, the method further includes: Acquire multiple speech signal samples; For each of the speech signal samples, analyzing the language expression of the speech signal sample to determine a first match between the language expression and each preset emotion type, wherein each preset emotion type includes: joy, sadness, anger, surprise, fear, calmness, disgust, love, anxiety, embarrassment, confusion, contempt, desire, relief and expectation; For each of the speech signal samples, analyzing the sound attribute of the speech signal sample to determine a second matching situation between the sound attribute and each of the preset emotion types; Determining the emotion type labels corresponding to each of the plurality of speech signal samples from the preset emotion types according to the first matching conditions and the second matching conditions corresponding to each of the plurality of speech signal samples; The CNN model and the RNN model are trained using the multiple speech signal samples and their corresponding emotion type labels.

[0007] As a possible implementation manner, analyzing the language expression of the speech signal sample to determine a first match between the language expression and each preset emotion type includes: A first matching situation between the language expression and each of the preset emotion types is determined according to the number of occurrences of each of the words associated with each of the preset emotion types in the language expression.

[0008] As a possible implementation manner, analyzing the sound attribute of the speech signal sample to determine the second matching situation between the sound attribute and each preset emotion type includes: The attribute features of the sound attributes associated with each of the preset emotion types are matched with the attribute features of the sound attributes of the speech signal sample to obtain a second matching situation between the sound attributes and the preset emotion types, wherein the sound attributes include: intonation and speech speed.

[0009] As a possible implementation, the CNN model includes: An input layer, used for receiving the time series data and a feature map composed of the acoustic features of the multiple dimensions; A convolution unit, used for extracting features from shallow features to deep features layer by layer on the feature map; One or more fully connected layers, used to process the features extracted by the convolution unit to obtain the target features; The output layer is used to output the target features.

[0010] As a possible implementation, the convolution unit includes: a plurality of convolution layers connected through activation function layers and pooling layers.

[0011] As a possible implementation, converting the speech signal into time series data and extracting acoustic features of multiple dimensions from the speech signal includes: Preprocessing the speech signal and converting the preprocessed speech signal into time series data, wherein the preprocessing includes: denoising, audio format standardization and sampling rate unification; Acoustic features of multiple dimensions are extracted from the preprocessed speech signal, wherein the acoustic features of multiple dimensions include: spectral features, Mel-frequency cepstral coefficients and spectrogram.

[0012] According to a second aspect of the present application, a speech emotion recognition device is provided, the device comprising: A signal acquisition module, used to acquire a voice signal from a user; A signal processing module, used for converting the speech signal into time series data, and extracting acoustic features of multiple dimensions from the speech signal, wherein the acoustic features are used for reflecting the physical properties of the speech signal; A feature extraction module, used to extract features from the time series data and the acoustic features of the multiple dimensions through a pre-trained convolutional neural network (CNN) model to obtain target features; The emotion recognition module is used to determine the recognition result of the emotion type according to the target feature through a pre-trained recurrent neural network (RNN) model.

[0013] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the speech emotion recognition method described in the first aspect are implemented.

[0014] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the speech emotion recognition method as described in the first aspect are implemented.

[0015] The embodiments of the present application include the following advantages: by introducing the CNN model, deep-level features (i.e., target features) are further extracted from the time series data and the acoustic features of the multiple dimensions, thereby avoiding the subjectivity and incompleteness of the manually designed feature extraction in the traditional method; and by introducing the RNN model, the recognition result of the emotion type is automatically determined according to the target features extracted by the CNN model, thereby realizing speech emotion recognition in the form of a hybrid model, thereby avoiding the limitations of using a single model, and the robustness of emotion classification can be improved by means of the adaptability and generalization ability of the model; in this way, the accuracy of speech emotion recognition can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 is an implementation flow chart of a speech emotion recognition method in an embodiment of the present application; Figure 2 is a schematic diagram of the architecture of a speech emotion recognition system in an embodiment of the present application; Figure 3 is a schematic diagram of an implementation process of a speech emotion recognition method in an embodiment of the present application; Figure 4 It is a structural diagram of a deep learning model in an embodiment of the present application; Figure 5 is a structural schematic diagram of a speech emotion recognition device in an embodiment of the present application; Figure 6 It is a schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0018] To facilitate understanding of the technical solutions provided by the present application, the main technical concepts involved in the embodiments of the present application are briefly described below.

[0019] Convolutional Neural Network (CNN): A neural network specifically designed to process data with a grid structure, such as images. CNN simulates the visual processing mechanism of the human brain and uses convolution operations to extract features from data, thereby achieving tasks such as classification and recognition of data such as images.

[0020] Recurrent Neural Network (RNN): is a neural network specifically designed to process sequence data. Unlike traditional neural networks (such as fully connected neural networks or convolutional neural networks), RNN can remember previous information and use this information to affect the output of subsequent nodes. This feature makes RNN perform well when processing time series data (such as text, video sequences, etc.).

[0021] Handcrafted Features: It is a commonly used technique in fields such as machine learning, computer vision, and signal processing. It refers to extracting feature representations that are useful for specific tasks from data through manual analysis and selection.

[0022] With the rapid development of artificial intelligence technology, speech emotion recognition has shown great application potential in many fields such as human-computer interaction, intelligent customer service and psychological counseling. Traditional speech emotion recognition methods mainly rely on manually designed feature extraction and classification algorithms, which has at least the following shortcomings: ① Lack of accuracy: Traditional speech emotion recognition methods often find it difficult to capture all the subtle emotional changes in speech signals; this is mainly because they rely on limited manually designed features, which are difficult to fully reflect the emotional information in the speech.

[0023] ② Weak generalization ability: Traditional speech emotion recognition methods have weak generalization ability, that is, they may not perform well in new environments or new data sets; this is mainly because the feature extraction and classification algorithm design of traditional methods often rely on specific data sets and environmental conditions. When faced with new, unseen speech samples, their recognition performance may drop significantly.

[0024] ③ Low efficiency: The processing efficiency of traditional speech emotion recognition methods may be low; this is mainly because the manually designed feature extraction and classification algorithms may have complex preprocessing steps and feature extraction processes, and require a lot of computing resources for emotion classification to ensure the accuracy of the results. These steps may increase the system's latency and power consumption, thereby limiting its performance in real-time applications.

[0025] ④ Poor flexibility: Traditional speech emotion recognition methods have poor flexibility, that is, they may be difficult to adapt to different application scenarios and changes in demand. This is mainly because they are often designed to be optimized for specific tasks and lack sufficient flexibility and scalability to cope with new application scenarios or changes in demand.

[0026] ⑤ Large data requirements: Traditional speech emotion recognition methods require a large amount of annotated data for algorithm design (or parameter determination of a single algorithm model) to ensure its recognition accuracy. However, in practical applications, obtaining high-quality annotated data is often a difficult task, which leads to insufficient or biased training data, thus affecting the accuracy and generalization ability of emotion recognition.

[0027] In response to the problems existing in the above-mentioned related technologies, the embodiments of the present application propose a speech emotion recognition method, device, equipment and medium. By introducing a hybrid model composed of a CNN model and a RNN model to perform speech emotion recognition, it can solve the problems of insufficient accuracy, weak generalization ability, low efficiency, poor flexibility and large data requirements existing in traditional speech emotion recognition methods based on manually designed feature extraction and classification algorithms, so that speech emotion recognition can be better applied in multiple fields such as human-computer interaction, intelligent customer service, psychological counseling, etc., thereby improving user experience.

[0028] In the following, in combination with the accompanying drawings, a speech emotion recognition method, apparatus, device and medium provided in the embodiments of the present application are described in detail through some embodiments and their application scenarios.

[0029] First, refer to Figure 1 As shown, it is a flow chart of an implementation of a method for speech emotion recognition provided by an embodiment of the present application, and the method comprises the following steps: Step S11: Acquire a voice signal from a user.

[0030] In specific implementation, voice signals input by users can be obtained through interfaces such as web pages, for example, voice signals in which users ask questions or provide feedback to the intelligent customer service system.

[0031] Step S12: converting the speech signal into time series data, and extracting acoustic features of multiple dimensions from the speech signal, wherein the acoustic features are used to reflect the physical properties of the speech signal.

[0032] In a specific implementation, the acquired speech signals are arranged in chronological order (and at a certain time interval) to obtain time series data suitable for processing by a deep learning model (such as a CNN model and an RNN model); and acoustic features of multiple dimensions are extracted from the speech signals. For example, acoustic features that can reflect the physical properties of the speech signals, such as fundamental frequency, short-time energy, resonance peaks, and Mel-Frequency Cepstral Coefficients (MFCC), can be extracted from the speech signals, so that the RNN model can combine the time series data and the acoustic features of the multiple dimensions to more accurately and comprehensively extract deep-level features related to emotions (i.e., the target features).

[0033] Step S13: extracting the time series data and the acoustic features of the multiple dimensions through a pre-trained convolutional neural network (CNN) model to obtain target features.

[0034] In the specific implementation, the target features extracted by the CNN model are input into the RNN model for sequence modeling and classification to identify the emotion type reflected by the speech signal, thereby realizing speech emotion recognition based on a hybrid model composed of the CNN model and the RNN model. It can be understood that this hybrid model can achieve better recognition performance than a single model.

[0035] Step S14: Determine the recognition result of the emotion type according to the target feature through a pre-trained recurrent neural network (RNN) model.

[0036] In specific implementation, the recognition results of the emotion type can be output to the user in the form of text or voice; alternatively, strategy decisions and adjustments can be made based on the recognition results of the emotion type. For example, based on the recognition results of the emotion type, the dialogue strategy of the intelligent customer service system can be adjusted to enable the intelligent customer service system to provide more considerate and personalized services, thereby improving the user experience.

[0037] By adopting the technical solution of the embodiment of the present application, by introducing the CNN model, deep-level features (i.e., target features) are further extracted from the time series data and the acoustic features of the multiple dimensions, thereby avoiding the subjectivity and incompleteness of the manually designed feature extraction in the traditional method; and then by introducing the RNN model, the recognition result of the emotion type is automatically determined according to the target features extracted by the CNN model, thereby realizing speech emotion recognition in the form of a hybrid model, avoiding the limitations of using a single model, and improving the robustness of emotion classification with the help of the adaptability and generalization ability of the model; in this way, the accuracy of speech emotion recognition can be effectively improved.

[0038] As a possible implementation, converting the speech signal into time series data and extracting acoustic features of multiple dimensions from the speech signal includes: Step S121: preprocessing the speech signal and converting the preprocessed speech signal into time series data, wherein the preprocessing includes: denoising, audio format standardization and sampling rate unification.

[0039] In specific implementation, for speech signal denoising, it is first necessary to estimate the characteristics of background noise. For example, noise estimation can be achieved by analyzing noise in inactive segments of speech signals (such as silent segments); then, based on the results of noise estimation, an appropriate algorithm is used to suppress noise. Thus, by removing the noise component in the speech signal, the clarity and recognizability of the speech can be improved.

[0040] Taking spectral subtraction as an example, based on the separability of noise signals and speech signals in the frequency domain, the noise spectrum is subtracted from the spectrum of the noisy speech signal to obtain a relatively pure speech spectrum, thereby achieving speech signal denoising.

[0041] For the standardization of the audio format of speech signals, firstly, a unified audio format (such as WAV) is determined according to the needs of subsequent processing, and then the audio files used to store speech signals are converted into the unified audio format using audio processing software or library (such as sox). Thus, by converting the collected audio files of different formats into a unified format, subsequent speech signal processing and analysis are facilitated.

[0042] For the uniform sampling rate of speech signals, first determine a suitable sampling rate (i.e., target sampling rate, such as 44.1kHz) according to the needs of subsequent processing; then use resampling technology to convert the current sampling rate of the speech signal to the target sampling rate. In the sampling rate conversion process, interpolation or decimation operations can be introduced to ensure the continuity and smoothness of the speech signal. Thus, speech signals with different sampling rates are converted to a uniform sampling rate to meet the requirements of subsequent processing algorithms.

[0043] Step S122: extracting acoustic features of multiple dimensions from the preprocessed speech signal, wherein the acoustic features of multiple dimensions include: spectral features, Mel-frequency cepstral coefficients and spectrogram.

[0044] Among them, the Mel-frequency cepstral coefficient is a widely used speech feature that can simulate the human ear's perception of frequency; the spectrogram is used to show how the frequency of the speech signal changes over time, which helps to capture dynamic features; the spectral feature can be obtained by converting the speech signal from the time domain signal to the frequency domain signal based on the short-time Fourier transform (STFT).

[0045] As a possible implementation, the method further includes: Step S21: Acquire multiple speech signal samples.

[0046] In specific implementation, refer to Figure 2 The schematic diagram of the architecture of the speech emotion recognition system shown in the figure can be used to execute the speech emotion recognition method provided in the embodiment of the present application.

[0047] Specifically, the data collection and preprocessing layer in the speech emotion recognition system first collects open source and public speech data on the Internet to obtain multiple speech signal samples, wherein the multiple speech signal samples should cover the various preset emotion types to ensure the diversity and balance of the samples; and then performs preprocessing operations such as denoising, audio format standardization and unified sampling rate on the collected multiple speech signal samples to improve data quality and consistency.

[0048] Step S22: For each of the voice signal samples, the language expression of the voice signal sample is analyzed to determine a first match between the language expression and each preset emotion type, wherein the preset emotion types include: joy, sadness, anger, surprise, fear, calmness, disgust, love, anxiety, embarrassment, confusion, contempt, desire, relief and expectation.

[0049] In specific implementation, the matching degree (such as matching score) between the language expression in the speech signal sample and each preset emotion type can be determined based on experience, thereby obtaining the first matching situation.

[0050] Alternatively, the first matching situation between the language expression and each preset emotion type may also be determined according to the number of occurrences of each word associated with each preset emotion type in the language expression, specifically: The more words associated with joy included (i.e., appearing) in the language expression, the higher the matching degree (e.g., matching score) of the language expression with the preset emotion type of joy can be determined. Among them, the words associated with joy can be words that express positive and bright emotions, such as "Great!" and "I'm so happy!"

[0051] The more words associated with sadness contained in the language expression, the higher the matching degree (such as matching score) of the language expression with the preset emotion type of sadness can be determined. The words associated with sadness can be words expressing loss, regret or pain, such as "I am so sad" and "This is really sad".

[0052] The more words associated with anger contained in the language expression, the higher the matching degree (such as matching score) of the language expression with the preset emotion type of anger can be determined. The words associated with anger can be words such as "How could you do this!", "I'm angry!", etc., which express emotions full of accusation, dissatisfaction or threat.

[0053] The more words associated with surprise contained in the language expression, the higher the matching degree (such as matching score) of the language expression with the preset emotion type of surprise can be determined. The words associated with surprise can be words expressing surprise such as "Wow!" and "Really?"

[0054] The more words associated with fear contained in the language expression, the higher the matching degree (such as matching score) between the language expression and the preset emotion type of fear can be determined. The words associated with fear can be words such as "I am afraid" and "so scary".

[0055] The more words associated with calm contained in the language expression, the higher the matching degree (such as matching score) of the language expression with the preset emotion type of calm. The words associated with calm can be words without strong emotional color (i.e., the emotions expressed are peaceful and stable).

[0056] The more words associated with disgust contained in the language expression, the higher the matching degree (such as matching score) of the language expression with the preset emotion type of disgust. The words associated with disgust may be words expressing dissatisfaction or disgust, such as "disgusting" and "unbearable".

[0057] The more words associated with love included in the language expression, the higher the matching degree (such as matching score) of the language expression with the preset emotion type of love can be determined. The words associated with love can be words such as "I love you" and "You are awesome" that express emotions full of tenderness, care and praise.

[0058] The more words associated with anxiety included in the language expression, the higher the matching degree (such as matching score) of the language expression with the preset emotion type of anxiety can be determined. Among them, the words associated with anxiety can be words such as "I am nervous" and "I am worried that something will go wrong" that express emotions including worry, anxiety or nervousness.

[0059] The more words associated with embarrassment included in the language expression, the higher the matching degree (such as matching score) of the language expression with the preset emotion type of embarrassment can be determined. The words associated with embarrassment can be words such as "sorry", "I didn't mean that" used to apologize, deny or change the subject (i.e., used to cover up the embarrassment emotion).

[0060] The more words associated with confusion included in the language expression, the higher the matching degree (such as matching score) of the language expression with the preset emotion type of confusion can be determined. The words associated with confusion can be words used for asking, repeating or expressing confusion, such as "what does this mean?" and "I don't quite understand".

[0061] The more words associated with contempt contained in the language expression, the higher the matching degree (such as matching score) of the language expression with the preset emotion type of contempt can be determined. The words associated with contempt can be words such as "this is ridiculous" and "you are such a fool" that express emotions that include derogation, ridicule or disdain.

[0062] The more words associated with desire contained in the language expression, the higher the matching degree (such as matching score) of the language expression with the preset emotion type of desire can be determined. The words associated with desire can be words such as "I really want that", "Can you give it to me?", etc., which express emotions including expectation, yearning or request.

[0063] The more words associated with relief included in the language expression, the higher the matching degree (such as matching score) of the language expression with the preset emotion type of relief can be determined. Among them, the words associated with relief can be words such as "finally over", "thank you for helping me", etc., which express emotions including relief, peace of mind or gratitude.

[0064] The more words associated with expectation included in the language expression, the higher the matching degree (such as matching score) of the language expression with the preset emotion type of expectation can be determined. Among them, the words associated with expectation can be words such as "I am looking forward to tomorrow" and "What do you think will happen?" that express emotions including expectation, excitement or curiosity about future events.

[0065] For each of the voice signal samples, the matching degree (such as matching score) between the determined language expression of the voice signal sample and each of the preset emotion types is summarized to obtain the first matching situation corresponding to the voice signal sample.

[0066] Step S23: for each of the speech signal samples, analyzing the sound attributes of the speech signal sample to determine a second matching situation between the sound attributes and each of the preset emotion types.

[0067] In a specific implementation, the sound attributes of the speech signal sample can be analyzed based on experience (for example, the emotional tension of the sound, the tendency of the expressed emotion, and other attributes can be analyzed based on experience) to determine the matching score between the sound attributes and the various preset emotion types, thereby obtaining the second matching situation.

[0068] Alternatively, the attribute features of the sound attributes associated with each of the preset emotion types may be matched with the attribute features of the sound attributes of the speech signal sample to obtain a second matching situation between the sound attributes and the preset emotion types, wherein the sound attributes include at least: intonation and speech speed, specifically: The higher the degree of match between the attribute characteristics of the sound attributes associated with joy: rising tone and slightly faster speaking speed (as well as vitality and enthusiasm in the voice, etc.) and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of match (such as matching score) between the sound attributes of the speech signal sample and the preset emotion type of joy.

[0069] The higher the degree of match between the attribute characteristics of the sound attributes associated with sadness: low voice tone and slow speaking speed (as well as trembling or choking in the voice, etc.) and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of match (such as matching score) between the sound attributes of the speech signal sample and the preset emotion type of sadness can be determined.

[0070] The higher the degree of match between the attribute characteristics of the sound attributes associated with anger: high-pitched tone and fast and rapid speaking speed (as well as strong emotional tension in the voice, etc.) and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of match (such as matching score) between the sound attributes of the speech signal sample and the preset emotion type of anger can be determined.

[0071] The higher the degree of match between the attribute characteristics of the sound attributes associated with surprise: a sudden rise or fall in tone and a brief pause in speech speed and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of match (such as a matching score) between the sound attributes of the speech signal sample and the preset emotion type of surprise.

[0072] The higher the degree of match between the attribute characteristics of the sound attributes associated with fear: trembling tone and faster or intermittent speaking speed (as well as a deep and nervous voice, etc.) and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of match (such as a matching score) between the sound attributes of the speech signal sample and the preset emotion type of fear.

[0073] The higher the degree of match between the attribute characteristics of the sound attributes associated with calmness: steady tone and moderate speaking speed (as well as clear and soft voice, etc.) and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of match (such as matching score) between the sound attributes of the speech signal sample and the preset emotion type of calmness can be determined.

[0074] The attribute characteristics of the sound attributes associated with disgust: the tone is slightly sarcastic or dissatisfied and the speaking speed varies due to emotions. The higher the degree of match with the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of match (such as matching score) between the sound attributes of the speech signal sample and the preset emotion type of disgust.

[0075] The higher the degree of match between the attribute characteristics of the sound attributes associated with love: soft tone and moderate speaking speed (and voice full of warmth and emotion, etc.) and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of match (such as matching score) between the sound attributes of the speech signal sample and the preset emotion type of love.

[0076] The higher the degree of match between the attribute characteristics of the sound attributes associated with anxiety: slightly hasty tone and slightly fast or unstable speaking speed (as well as a sense of nervousness in the voice, etc.) and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of match (such as matching score) between the sound attributes of the speech signal sample and the preset emotion type of anxiety can be determined.

[0077] The higher the degree of match between the attribute characteristics of the sound attributes associated with embarrassment: soft or low tone (and a hint of discomfort in the voice, etc.) and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of match (such as the matching score) between the sound attributes of the speech signal sample and the preset emotion type of embarrassment can be determined.

[0078] The higher the degree of match between the attribute characteristics of the sound attributes associated with confusion: rising intonation (to express doubt) and slowing speech speed (to seek more information) and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of match (such as matching score) between the sound attributes of the speech signal sample and the preset emotion type of confusion can be determined.

[0079] The higher the degree of matching between the attribute characteristics of the sound attributes associated with contempt: the tone of voice with a sarcastic or contemptuous tone and the steady speaking speed (and the tone of voice revealing dissatisfaction, etc.) and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of matching (such as the matching score) between the sound attributes of the speech signal sample and the preset emotion type of contempt.

[0080] The higher the degree of matching between the attribute characteristics of the sound attributes associated with desire: a slightly anxious or expectant tone and a slightly faster speaking speed (to express inner desire) and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of matching (such as a matching score) between the sound attributes of the speech signal sample and the preset emotion type of desire can be determined.

[0081] The higher the degree of match between the attribute characteristics of the sound attributes associated with comfort: soft and relaxed tone and slow speaking speed (to express inner peace) and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of match (such as matching score) between the sound attributes of the speech signal sample and the preset emotion type of comfort can be determined.

[0082] The higher the degree of matching between the attribute characteristics of the sound attributes associated with expectation: rising tone with excitement and slightly faster speaking speed (to express inner excitement) and the attribute characteristics of the sound attributes of the speech signal sample, the higher the degree of matching (such as matching score) between the sound attributes of the speech signal sample and the preset emotion type of expectation can be determined.

[0083] For each of the speech signal samples, the attribute characteristics of the sound attributes of the speech signal sample can be identified according to a pre-trained deep learning model, and then the degree of match between the sound attributes of the speech signal sample and the preset emotion types is determined (for example, the matching score can be obtained by the weighted sum of the matching scores corresponding to different attribute characteristics). After that, the obtained matching degrees are summarized to obtain the second matching situation corresponding to the speech signal sample.

[0084] Step S24: determining the emotion type labels corresponding to each of the plurality of speech signal samples from the preset emotion types according to the first matching conditions and the second matching conditions corresponding to each of the plurality of speech signal samples.

[0085] In specific implementation, the first matching situation and the second matching situation corresponding to each of the multiple voice signal samples can be comprehensively considered by calculating the weighted sum, taking the intersection, etc., so as to obtain the preset emotion type with the highest matching degree for each of the multiple voice signal samples, and then the preset emotion types with the highest matching degree obtained can be used as the emotion type labels corresponding to each of the multiple voice signal samples.

[0086] Step S25: Use the multiple speech signal samples and their corresponding emotion type labels to train the CNN model and the RNN model.

[0087] In the specific implementation, the acoustic feature extraction layer in the speech emotion recognition system first uses traditional speech signal processing methods to extract acoustic features of multiple dimensions from the multiple speech signal samples, such as fundamental frequency, short-time energy, resonance peaks and Mel-frequency cepstral coefficients, etc., and converts the multiple speech signal samples into a format suitable for deep learning model processing: time series data format.

[0088] The emotion recognition model layer in the system uses the acoustic features and time series obtained by processing the multiple voice signal samples by the acoustic feature extraction layer, as well as the emotion type labels corresponding to each of the multiple voice signal samples, to train the hybrid model composed of the CNN model and the RNN model, and when the training conditions are met (such as the number of iterations is reached or the model performance meets the requirements), the training of the hybrid model is terminated to obtain the trained CNN model and RNN model.

[0089] Reference Figure 3 The schematic diagram of the implementation process of the speech emotion recognition method shown in the figure, in the subsequent model inference stage, the emotion recognition model layer extracts the emotion features (i.e., target features) in the speech signal through the CNN model, and then transmits the extracted emotion features to the RNN model to identify the emotion type. The result output layer in the speech emotion recognition system then outputs the emotion type recognition result in the form of text or voice to the user or application system such as intelligent customer service, so that the user or application system can adjust the next strategy according to the recognition result.

[0090] In this embodiment, the hybrid model composed of the CNN model and the RNN model is trained using a labeled emotion classification data set, so that the hybrid model learns how to map the input data to a preset emotion type. Compared with training the CNN model and the RNN model separately, this training method based on the hybrid model can reduce the workload of data labeling; and because the hybrid model can often achieve better recognition performance than a single model, this training method based on the hybrid model can also effectively reduce the amount of data required for model training convergence.

[0091] As a possible implementation, the CNN model includes: An input layer, used for receiving a feature graph composed of the time series data and the acoustic features of the multiple dimensions, wherein different rows and different columns of the feature graph correspond to different time points and different data dimensions respectively; A convolution unit, used for extracting features from shallow features to deep features layer by layer on the feature map; One or more fully connected layers, used to process the features extracted by the convolution unit to obtain the target features; The output layer is used to output the target features.

[0092] In a specific implementation, the convolution unit can stack multiple convolution layers to achieve layer-by-layer feature extraction from shallow features to deep features, that is, to achieve layer-by-layer feature extraction from low-level features (extracted by shallow convolution layers) to high-level features (which can be more complex speech patterns such as syllables and morphemes, which are extracted by deep convolution layers).

[0093] One or more fully connected layers may be connected after the convolution unit to further process (and classify) the high-level features extracted by the convolution unit, such as fusing the features extracted by different convolution kernels in the convolution unit to form a more comprehensive feature representation. This feature fusion helps to capture multiple characteristics in the speech signal, thereby improving the accuracy of subsequent recognition or classification.

[0094] Optionally, the convolution unit includes: a plurality of convolution layers connected through an activation function layer and a pooling layer.

[0095] Among them, the convolution layer captures local features in the speech signal through local connections and weight sharing mechanisms, that is, the convolution layer uses the convolution kernel to slide on the input feature map and perform convolution operations with the local area to generate a new feature map; the pooling layer reduces the size of the feature map by downsampling while retaining important feature information, which helps to reduce the amount of calculation and improve the robustness of the model. The pooling layer can be used to perform maximum pooling operations or average pooling operations.

[0096] In a specific implementation, the output of a convolutional layer is processed by an activation function layer and a pooling layer respectively, and the output of the activation function layer and the output of the pooling layer are then processed by the next convolutional layer, thereby connecting multiple convolutional layers through the activation function layer and the pooling layer to form a convolutional unit.

[0097] In this embodiment, by introducing a combination of multiple convolutional layers, pooling layers, and activation function layers in the CNN model, it is ensured that the model can gradually extract higher-level features.

[0098] Optionally, the activation function layer may use a nonlinear activation function such as a rectified linear unit (ReLU) to increase the nonlinearity of the model to help extract more complex features.

[0099] Taking the example of embedding the above-mentioned speech emotion recognition system into an intelligent customer service system, the implementation and application process of the speech emotion recognition method provided in the embodiment of the present application are illustrated.

[0100] When the user interacts with the customer service system through voice, the voice emotion recognition system first receives the voice signal input by the user and performs preprocessing, acoustic feature extraction, and time series data conversion. Specifically, after receiving the voice signal, the voice emotion recognition system first performs noise reduction processing to remove background noise and irrelevant noise, then removes silent segments to reduce the amount of calculation, and finally unifies the sampling rate to ensure data consistency, thereby completing the preprocessing. Then, the preprocessed voice signal is subjected to acoustic feature extraction methods such as MFCC to extract key features that can reflect the emotional state of the voice (i.e., acoustic features of multiple dimensions), and the voice signal is converted into time series data.

[0101] Then, the speech emotion recognition system uses the CNN model to extract features from the feature graph composed of time series data and acoustic features of multiple dimensions, and then uses the RNN model to identify the emotion type based on the features extracted by the CNN model. Specifically, the system inputs the feature graph into a pre-trained deep learning model (i.e., a hybrid model composed of a CNN model and an RNN model). The model can identify the emotion type in the user's voice signal through learning and analysis of a multi-layer neural network; for example, if the model analyzes that the user's tone is rapid and the pitch is high, the recognition result of the emotion type "anger" can be obtained. The structural diagram of the deep learning model is shown in the figure below. Figure 4 shown.

[0102] Finally, the intelligent customer service system outputs the recognition results determined by the speech emotion recognition system to the customer service staff in the form of text or automatically triggers the corresponding service process or adjusts the response strategy. For example, if the user shows anger, the intelligent customer service system can adopt a more gentle and patient response; if the user shows happiness or satisfaction, the intelligent customer service system can provide a more positive and affirmative response. In addition, the speech emotion recognition system can also compare the recognition results of the emotion type with the actual feedback of the user to continuously optimize the parameters and performance of the deep learning model and improve the accuracy and robustness of speech emotion recognition. The experimental results show that the speech emotion recognition system has excellent performance in various emotion recognition tasks, significantly improving the user experience of the intelligent customer service system.

[0103] It should be noted that the speech emotion recognition method provided in the embodiments of the present application has at least the following application extensions: 1. Improved human-computer interaction experience: In the fields of smart customer service, smart home, and virtual reality, accurate identification of the user's emotional state (i.e., emotional type) helps provide users with more intimate and personalized services during human-computer interaction, thereby improving the user experience.

[0104] 2. Mental health assistance: In the fields of psychological counseling and psychotherapy, by accurately identifying the user's emotional state, it can assist psychologists in understanding the patient's emotional changes, thereby providing strong support for clinical diagnosis and treatment.

[0105] 3. Polygraph and security detection: In the fields of justice, security, etc., accurate identification of the user's emotional state can assist in determining the psychological state of the person being tested, thereby improving the accuracy and efficiency of polygraph and security detection.

[0106] Based on the above examples and embodiments, the speech emotion recognition method provided in this application can at least achieve the following optimization effects: 1. Improve recognition accuracy: By introducing the CNN model to automatically extract deep features from speech signals, the subjectivity and incompleteness of manual feature extraction in traditional methods can be reduced, thereby improving the accuracy of speech emotion recognition.

[0107] 2. Enhanced generalization capability: Deep learning models such as CNN models and RNN models can learn more general feature representations, thereby being able to process speech signals from different sources and of different qualities, thereby improving the robustness of the system and making the system's recognition performance more stable and generalizable in different environments and with different speakers.

[0108] 3. Optimize the algorithm structure: Using a hybrid model for training and reasoning can reduce the computational complexity of the model, improve recognition speed, and reduce resource consumption. By combining model-related hardware acceleration technology, real-time or near real-time speech emotion recognition functions can be achieved to meet actual application needs.

[0109] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0110] Second, Figure 5 : is a structural schematic diagram of a speech emotion recognition device according to an embodiment of the present application, the device comprising: A signal acquisition module 510 is used to acquire a voice signal from a user; A signal processing module 520, configured to convert the speech signal into time series data, and extract acoustic features of multiple dimensions from the speech signal, wherein the acoustic features are used to reflect the physical properties of the speech signal; A feature extraction module 530 is used to extract features from the time series data and the acoustic features of the multiple dimensions through a pre-trained convolutional neural network (CNN) model to obtain target features; The emotion recognition module 540 is used to determine the recognition result of the emotion type according to the target feature through a pre-trained recurrent neural network (RNN) model.

[0111] By adopting the technical solution of the embodiment of the present application, by introducing the CNN model, deep-level features (i.e., target features) are further extracted from the time series data and the acoustic features of the multiple dimensions, thereby avoiding the subjectivity and incompleteness of the manually designed feature extraction in the traditional method; and then by introducing the RNN model, the recognition result of the emotion type is automatically determined according to the target features extracted by the CNN model, thereby realizing speech emotion recognition in the form of a hybrid model, avoiding the limitations of using a single model, and improving the robustness of emotion classification with the help of the adaptability and generalization ability of the model; in this way, the accuracy of speech emotion recognition can be effectively improved.

[0112] Optionally, the device further comprises a model training module, configured to perform the following steps: Acquire multiple speech signal samples; For each of the speech signal samples, analyzing the language expression of the speech signal sample to determine a first match between the language expression and each preset emotion type, wherein each preset emotion type includes: joy, sadness, anger, surprise, fear, calmness, disgust, love, anxiety, embarrassment, confusion, contempt, desire, relief and expectation; For each of the speech signal samples, analyzing the sound attribute of the speech signal sample to determine a second matching situation between the sound attribute and each of the preset emotion types; Determining the emotion type labels corresponding to each of the plurality of speech signal samples from the preset emotion types according to the first matching conditions and the second matching conditions corresponding to each of the plurality of speech signal samples; The CNN model and the RNN model are trained using the multiple speech signal samples and their corresponding emotion type labels.

[0113] Optionally, the model training module is further used to determine a first match between the language expression and each preset emotion type according to the number of occurrences of each word associated with each preset emotion type in the language expression.

[0114] Optionally, the model training module is also used to match the attribute characteristics of the sound attributes associated with each of the preset emotion types with the attribute characteristics of the sound attributes of the speech signal sample, respectively, to obtain a second matching situation between the sound attributes and the preset emotion types, the sound attributes including: intonation and speaking speed.

[0115] Optionally, the CNN model includes: An input layer, used for receiving the time series data and a feature map composed of the acoustic features of the multiple dimensions; A convolution unit, used for extracting features from shallow features to deep features layer by layer on the feature map; One or more fully connected layers, used to process the features extracted by the convolution unit to obtain the target features; The output layer is used to output the target features.

[0116] Optionally, the convolution unit includes: a plurality of convolution layers connected through an activation function layer and a pooling layer.

[0117] Optionally, the signal processing module 520 is further configured to perform the following steps: Preprocessing the speech signal and converting the preprocessed speech signal into time series data, wherein the preprocessing includes: denoising, audio format standardization and sampling rate unification; Acoustic features of multiple dimensions are extracted from the preprocessed speech signal, wherein the acoustic features of multiple dimensions include: spectral features, Mel-frequency cepstral coefficients and spectrogram.

[0118] It should be noted that the device embodiment is similar to the method embodiment, so the description is relatively simple, and the relevant parts can be referred to the method embodiment.

[0119] The present application also provides an electronic device, referring to Figure 6 , Figure 6 Schematic diagram of an electronic device proposed in an embodiment of the present application. Figure 6 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement the steps in the speech emotion recognition method disclosed in the embodiment of the present application.

[0120] The embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the speech emotion recognition method disclosed in the embodiment of the present application is implemented.

[0121] The embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the speech emotion recognition method disclosed in the embodiment of the present application.

[0122] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0123] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, systems, devices, storage media, and program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0127] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0128] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0129] The above is a detailed introduction to the speech emotion recognition method, device, equipment and medium provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A speech emotion recognition method, characterized in that: The method comprises: Acquire a voice signal from a user; Converting the speech signal into time series data, and extracting acoustic features of multiple dimensions from the speech signal, wherein the acoustic features are used to reflect the physical properties of the speech signal; Extracting features of the time series data and the acoustic features of the multiple dimensions through a pre-trained convolutional neural network (CNN) model to obtain target features; The recognition result of the emotion type is determined according to the target feature through a pre-trained recurrent neural network (RNN) model.

2. The method according to claim 1, characterized in that The method further comprises: Acquire multiple speech signal samples; For each of the speech signal samples, analyzing the language expression of the speech signal sample to determine a first match between the language expression and each preset emotion type, wherein each preset emotion type includes: joy, sadness, anger, surprise, fear, calmness, disgust, love, anxiety, embarrassment, confusion, contempt, desire, relief and expectation; For each of the speech signal samples, analyzing the sound attribute of the speech signal sample to determine a second matching situation between the sound attribute and each of the preset emotion types; Determining the emotion type labels corresponding to each of the plurality of speech signal samples from the preset emotion types according to the first matching conditions and the second matching conditions corresponding to each of the plurality of speech signal samples; The CNN model and the RNN model are trained using the multiple speech signal samples and their corresponding emotion type labels.

3. The method according to claim 2, characterized in that The analyzing the language expression of the speech signal sample to determine a first matching situation between the language expression and each preset emotion type includes: A first matching situation between the language expression and each of the preset emotion types is determined according to the number of occurrences of each of the words associated with each of the preset emotion types in the language expression.

4. The method according to claim 2, characterized in that: The analyzing the sound attribute of the speech signal sample to determine the second matching situation between the sound attribute and each preset emotion type includes: The attribute features of the sound attributes associated with each of the preset emotion types are matched with the attribute features of the sound attributes of the speech signal sample to obtain a second matching situation between the sound attributes and the preset emotion types, wherein the sound attributes include: intonation and speech speed.

5. The method according to claim 1, characterized in that The CNN model includes: An input layer, used for receiving the time series data and a feature map composed of the acoustic features of the multiple dimensions; A convolution unit, used for extracting features from shallow features to deep features layer by layer on the feature map; One or more fully connected layers, used to process the features extracted by the convolution unit to obtain the target features; The output layer is used to output the target features.

6. The method according to claim 5, characterized in that The convolution unit includes: a plurality of convolution layers connected through an activation function layer and a pooling layer.

7. The method according to any one of claims 1 to 6, characterized in that: The step of converting the speech signal into time series data and extracting acoustic features of multiple dimensions from the speech signal comprises: Preprocessing the speech signal and converting the preprocessed speech signal into time series data, wherein the preprocessing includes: denoising, audio format standardization and sampling rate unification; Acoustic features of multiple dimensions are extracted from the preprocessed speech signal, wherein the acoustic features of multiple dimensions include: spectral features, Mel-frequency cepstral coefficients and spectrogram.

8. A speech emotion recognition device, characterized in that: The device comprises: A signal acquisition module, used to acquire a voice signal from a user; A signal processing module, used for converting the speech signal into time series data, and extracting acoustic features of multiple dimensions from the speech signal, wherein the acoustic features are used for reflecting the physical properties of the speech signal; A feature extraction module, used to extract features from the time series data and the acoustic features of the multiple dimensions through a pre-trained convolutional neural network (CNN) model to obtain target features; The emotion recognition module is used to determine the recognition result of the emotion type according to the target feature through a pre-trained recurrent neural network (RNN) model.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the speech emotion recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the speech emotion recognition method as described in any one of claims 1 to 7 is implemented.

Images