Voice emotion recognition method and device, electronic equipment, and storage medium

Abstract

The voice emotion recognition method and device, the electronic device and the storage medium provided by the embodiment of the present application relate to the technical field of artificial intelligence and the field of financial technology. The method comprises the following steps: obtaining a target voice signal; performing acoustic spectrum feature extraction on the target voice signal to obtain a target acoustic spectrum feature vector; performing emotion feature transformation on the target acoustic spectrum feature vector through a pre-trained target emotion feature extractor to obtain a target emotion latent feature vector; performing feature sampling on the target emotion latent feature vector to obtain at least two target emotion sampling feature vectors; performing feature selection on each target emotion sampling feature vector to obtain a target key emotion feature vector; and performing emotion classification on the target key emotion feature vector through a pre-trained target classifier to obtain a target emotion category. The embodiment of the present application can improve the emotion recognition accuracy and the recognition efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and the technical field of financial technology, and in particular relates to a speech emotion recognition method and device, an electronic device and a storage medium. BACKGROUND

[0002] Speech emotion recognition is a technology for analyzing and recognizing the emotional state of a speaker from a speech signal. The prior art proposes a method based on deep learning, which needs to learn emotional features directly from the speech signal and perform emotion classification. However, it is difficult to learn emotional features directly from the speech signal, and usually requires a large amount of computing resources, and the accuracy still has a large room for improvement.

[0003] Therefore, how to provide a speech emotion recognition method capable of improving emotion recognition accuracy and improving recognition efficiency has become a technical problem to be solved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a speech emotion recognition method and device, an electronic device and a storage medium, which can improve emotion recognition accuracy and improve recognition efficiency.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present application provides a speech emotion recognition method, which comprises:

[0006] obtaining a target speech signal;

[0007] performing acoustic spectrum feature extraction on the target speech signal to obtain a target acoustic spectrum feature vector;

[0008] performing emotion feature transformation on the target acoustic spectrum feature vector by a pre-trained target emotion feature extractor to obtain a target emotion latent feature vector;

[0009] performing feature sampling on the target emotion latent feature vector to obtain at least two target emotion sampling feature vectors;

[0010] performing feature selection on each target emotion sampling feature vector to obtain a target key emotion feature vector;

[0011] performing emotion classification on the target key emotion feature vector by a pre-trained target classifier to obtain a target emotion category.

[0012] In some embodiments, before the pre-trained target emotion feature extractor performs feature transformation on the target acoustic spectrum feature vector to obtain a target emotion latent feature vector, the method further comprises:

[0013] training the target emotion feature extractor, specifically comprising:

[0014] obtaining a spectrum feature of the speech sample signal to obtain a sample spectrum feature vector;

[0015] performing feature transformation on the sample spectrum feature vector by a preset initial emotion feature extractor to obtain a sample emotion latent feature vector;

[0016] sampling the sample emotion latent feature vector to obtain at least two sample emotion sampling feature vectors;

[0017] decoding each of the emotion sampling feature vectors to obtain a speech reconstruction feature vector; wherein a dimension of the speech reconstruction feature vector is the same as a dimension of the sample spectrum feature vector;

[0018] performing loss reconstruction according to the speech sample signal and the speech reconstruction feature vector to obtain target loss data;

[0019] adjusting parameters of the initial emotion feature extractor according to the target loss data to obtain a target emotion feature extractor.

[0020] In some embodiments, before the adjusting parameters of the initial emotion feature extractor according to the target loss data to obtain the target emotion feature extractor, the method further comprises:

[0021] updating the target loss data, specifically comprising:

[0022] performing feature selection on each of the sample emotion sampling feature vectors to obtain a sample key emotion feature vector;

[0023] performing emotion classification on the sample key emotion feature vector by a preset initial classifier to obtain a sample emotion category;

[0024] performing loss calculation according to the sample emotion category and a preset first emotion category label to obtain first category loss data; wherein the first emotion category label is a single label, and the first emotion category label indicates an emotion category to which the speech sample signal belongs;

[0025] updating the target loss data according to the first category loss data.

[0026] In some embodiments, before the performing emotion classification on the target key emotion feature vector by the pre-trained target classifier to obtain a target emotion category, the method further comprises:

[0027] training the target classifier, specifically comprising:

[0028] perform loss calculation according to the sample emotion category and a preset second emotion category label, to obtain second category loss data; wherein the second emotion category label is a multi-label, and the second emotion category label indicates a sentiment category to which the voice sample signal belongs and a sentiment degree of the sentiment category;

[0029] perform parameter adjustment on the initial classifier according to the second category loss data, to obtain the target classifier.

[0030] In some embodiments, the target emotion feature extractor includes a target convolutional layer and a gated recurrent unit network layer;

[0031] The emotion feature transformation on the target spectrogram feature vector by the pre-trained target emotion feature extractor includes:

[0032] convolutional processing on the target spectrogram feature vector by the target convolutional layer, to obtain an emotion initial feature vector;

[0033] context feature extraction on the emotion initial feature vector by the gated recurrent unit network layer, to obtain the target emotion latent feature vector.

[0034] In some embodiments, the target convolutional layer includes a first convolutional layer, a second convolutional layer, and a flattening layer, and a convolution kernel size of the first convolutional layer is smaller than a convolution kernel size of the second convolutional layer;

[0035] The convolutional processing on the target spectrogram feature vector by the target convolutional layer, to obtain an emotion initial feature vector, includes:

[0036] first convolution on the target spectrogram feature vector by the first convolutional layer, to obtain a first emotion initial feature vector;

[0037] second convolution on the first emotion initial feature vector by the second convolutional layer, to obtain a second emotion initial feature vector;

[0038] vector one-dimensionization on the second emotion initial feature vector by the flattening layer, to obtain the emotion initial feature vector.

[0039] In some embodiments, the feature selection on each of the target emotion sampling feature vectors, to obtain a target key emotion feature vector, includes:

[0040] correlation calculation on each of the target emotion sampling feature vectors and other target emotion sampling feature vectors, to obtain a target emotion feature correlation;

[0041] According to the target emotion feature correlation, weights of each target emotion sample feature vector are determined to obtain target weights;

[0042] According to the target weights, the target key emotion feature vector is extracted from each target emotion sample feature vector.

[0043] To achieve the above object, a second aspect of the embodiment of the present application provides a speech emotion recognition device, which comprises:

[0044] A speech acquisition module is configured to acquire a target speech signal.

[0045] A feature extraction module is configured to perform acoustic spectrum feature extraction on the target speech signal to obtain a target acoustic spectrum feature vector.

[0046] A feature transformation module is configured to perform emotion feature transformation on the target acoustic spectrum feature vector by using a pre-trained target emotion feature extractor to obtain a target emotion latent feature vector.

[0047] A feature sampling module is configured to perform feature sampling on the target emotion latent feature vector to obtain at least two target emotion sample feature vectors.

[0048] A feature selection module is configured to perform feature selection on each target emotion sample feature vector to obtain a target key emotion feature vector.

[0049] An emotion classification module is configured to perform emotion classification on the target key emotion feature vector by using a pre-trained target classifier to obtain a target emotion category.

[0050] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech emotion recognition method of the first aspect when executing the computer program.

[0051] To achieve the above object, a fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the speech emotion recognition method of the first aspect.

[0052] The voice emotion recognition method, the voice emotion recognition device, the electronic device, and the storage medium provided in the application, when a target emotion category needs to be recognized from a target voice signal, first, a sound spectrum feature of the target voice signal is extracted to obtain a target sound spectrum feature vector, and then the target sound spectrum feature vector is transformed to obtain a target emotion latent feature vector. In this way, instead of directly extracting an emotion-related feature vector from the target voice signal, the sound spectrum feature is first extracted, and then the emotion feature is transformed, which can extract the emotion-related feature vector from the target voice signal, and the difficulty of extraction is relatively low. Considering that the target emotion latent feature vector contains feature information with different degrees of influence on emotion classification, at least two target emotion sampling feature vectors are first obtained by feature sampling, and then a target key emotion feature vector is obtained by feature selection. In this way, the feature vector with a relatively high degree of influence on emotion classification can be extracted from the target voice signal. Then, the target classifier is used for emotion classification to obtain the target emotion category. In summary, the embodiment reduces the difficulty of feature extraction when extracting features for emotion classification from the target voice signal, can save computing resources, and also ensures the accuracy of the features. In summary, the embodiment can improve the emotion recognition accuracy and improve the recognition efficiency.

[0053] Additional aspects and advantages of the application will be set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the following description and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a flowchart of the voice emotion recognition method provided by the embodiment of the application;

[0055] Figure 2 is a flowchart of the voice emotion recognition method provided by another embodiment of the application;

[0056] Figure 3 is a flowchart of the voice emotion recognition method provided by another embodiment of the application;

[0057] Figure 4 is Figure 1 a flowchart of step 103 in

[0058] Figure 5 is a flowchart of step 401 in Figure 4

[0059] Figure 6 is a flowchart of step 105 in Figure 1

[0060] Figure 7 is a flowchart of the voice emotion recognition method provided by another embodiment of the application;

[0061] ​​Figure 8 is a module structure block diagram of a voice emotion recognition device provided by an embodiment of the present application;

[0062] Figure 9 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0064] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0066] First, the meanings of several terms involved in the present application are analyzed:

[0067] Artificial intelligence (AI): is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, and artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] Natural language processing (NLP): NLP uses computers to process, understand, and use human languages (such as Chinese, English, etc.), and NLP is a branch of artificial intelligence, an interdisciplinary subject of computer science and linguistics, and is also commonly referred to as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, and discourse understanding. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and linguistic research related to language computing.

[0069] Voice emotion recognition technology: a technology that analyzes and recognizes the emotional state of a speaker from a voice signal. It can be used to improve the naturalness and friendliness of human-computer interaction, and provide more emotional information and services. The main task of voice emotion recognition is to convert voice signals into corresponding emotion categories, such as anger, happiness, sadness, and neutrality.

[0070] In the field of financial technology, voice emotion recognition technology has many application scenarios, for example: (1) Customer emotion analysis: financial institutions can use voice emotion recognition technology to analyze the emotions in customer calls. By monitoring and analyzing the voice emotions of customers in real time, financial institutions can better understand the attitudes and emotions of customers, and take appropriate measures based on the recognition results, such as adjusting customer service strategies or guiding customer investment directions. (2) Voice assistants and emotional interactions: embedding voice assistants and emotional interaction functions in financial services can provide users with personalized service experiences. Voice assistants can analyze user voice emotions to understand user emotional needs and provide more personalized financial recommendations and services. (3) Anti-fraud detection: voice emotion recognition can be used to detect anti-fraud behavior. By analyzing the voice emotions in phone transactions or customer service conversations, financial institutions can identify abnormal emotional patterns, fearful or nervous voice features, and improve the recognition of anti-fraud behavior.

[0071] However, the related technology has difficulty in directly learning emotional features from voice signals using neural network models, often requiring a large amount of computing resources, and the accuracy still has a lot of room for improvement.

[0072] Therefore, the embodiments of the present application propose a voice emotion recognition method and a voice emotion recognition device, an electronic device, and a storage medium, which can improve the accuracy of emotion recognition and improve the recognition efficiency.

[0073] The voice emotion recognition method provided by the embodiments of the present application can be applied to a terminal and a server end, and can also be software running in the server end. The server end can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; the software can be an application that implements the voice emotion recognition method, but is not limited to the above forms.

[0074] The present application can be used in many general or special computer system environments or configurations. For example: server computers, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0075] The embodiments of the present application provide a voice emotion recognition method and a voice emotion recognition device, an electronic device, and a storage medium, which are specifically described by the following embodiments. First, the voice emotion recognition method in the embodiments of the present application is described.

[0076] It should be noted that in each specific embodiment of the present application, when it is necessary to perform relevant processing according to the voice data of the user and other data related to the identity or characteristics of the user, the permission or consent of the user is obtained first, and the collection, use, and processing of the data comply with relevant laws, regulations, and standards.

[0077] Reference Figure 1 , Figure 1 is an optional flowchart of the voice emotion recognition method provided by the embodiments of the present application, which can include but is not limited to steps 101 to 106.

[0078] Step 101, obtaining a target voice signal;

[0079] Step 102, performing acoustic spectrum feature extraction on the target voice signal to obtain a target acoustic spectrum feature vector;

[0080] In step 103, the target emotion feature extractor is used to perform emotion feature transformation on the target spectrum feature vector to obtain a target emotion latent feature vector.

[0081] In step 104, feature sampling is performed on the target emotion latent feature vector to obtain at least two target emotion sampling feature vectors.

[0082] In step 105, feature selection is performed on each target emotion sampling feature vector to obtain a target key emotion feature vector.

[0083] In step 106, the target key emotion feature vector is classified by the pre-trained target classifier to obtain a target emotion category.

[0084] The steps 101 to 106 shown in the embodiments of the present application, when facing the need to identify the target emotion category from the target speech signal, first perform spectrum feature extraction on the target speech signal to obtain a target spectrum feature vector, and then perform feature transformation on the target spectrum feature vector to obtain a target emotion latent feature vector. In this way, instead of directly extracting the target emotion latent feature vector from the target speech signal, the method of first performing spectrum feature extraction and then performing emotion feature transformation can extract a feature vector related to emotion from the target speech signal with relatively low difficulty. Considering that the target emotion latent feature vector contains feature information with different degrees of influence on emotion classification, at least two target emotion sampling feature vectors are obtained by feature sampling, and then a target key emotion feature vector is obtained by feature selection. In this way, the feature information with a relatively high degree of influence on emotion classification is extracted from the target speech signal. Then, the target classifier is used for emotion classification to obtain a target emotion category. In summary, the embodiments of the present application reduce the difficulty of feature extraction when extracting features for emotion classification from the target speech signal, can save computing resources, and also ensure the accuracy of the features. In summary, the embodiments of the present application can improve the accuracy of emotion recognition and improve the recognition efficiency.

[0085] In step 101 of some embodiments, a target speech signal is obtained. The target speech signal is used to represent the speech uttered by a certain speaker. Specifically, when the speech emotion recognition method is applied in a terminal, the target speech signal can be obtained by recording, Bluetooth transmission, wired transmission, or downloading. When the target speech signal is obtained by recording, the terminal is correspondingly configured with a microphone, and audio acquisition is performed through the microphone to realize recording of the target speech signal. When the speech emotion recognition method is applied in a server, the speech sample data can be uploaded to the server by a terminal, or downloaded from other servers or databases by the server.

[0086] In step 102 of some embodiments, a sound spectrum feature of the target voice signal is extracted to obtain a target sound spectrum feature vector. The sound spectrum feature refers to the distribution characteristics of sound in the frequency domain, which is usually displayed through a sound spectrum diagram. The sound spectrum diagram reflects the frequency components and their intensity of sound, which can help identify and analyze the characteristics of sound, such as pitch, tone, volume, and frequency components. For example, the sound spectrum feature is the Mel Frequency Cepstral Coefficients (MFCC), which can reflect the perceptual characteristics of human beings to sound.

[0087] The MFCC feature is calculated by the following steps: first, pre-emphasizing and framing the target voice signal; then, Hamming windowing and fast Fourier transform are performed on each frame; then, Mel filter bank filtering and logarithmic operation are performed on the spectrum; finally, discrete cosine transform is performed on the filter bank output to obtain the MFCC coefficients. In addition, other acoustic features such as the fundamental frequency (F0) and energy (E) of the voice signal are extracted and spliced together with the MFCC feature to form a more rich feature vector. In this embodiment, the sampling rate can be set to 16000 Hz, the frame length is 25 ms, the frame shift is 10 ms, the number of Mel filter banks is 40, and the number of MFCC coefficients is 13.

[0088] In an embodiment, with reference to Figure 2 Before step 103, the voice emotion recognition method provided in this embodiment can further include: training a target emotion feature extractor, specifically including:

[0089] Step 201: obtaining a sound spectrum feature of a voice sample signal to obtain a sample sound spectrum feature vector;

[0090] Step 202: performing feature transformation on the sample sound spectrum feature vector through a preset initial emotion feature extractor to obtain a sample emotion latent feature vector;

[0091] Step 203: sampling the sample emotion latent feature vector to obtain at least two sample emotion sampling feature vectors;

[0092] Step 204: decoding each emotion sampling feature vector to obtain a voice reconstruction feature vector;

[0093] Step 205: loss reconstruction is performed according to the voice sample signal and the voice reconstruction feature vector to obtain target loss data;

[0094] Step 206: parameter adjustment is performed on the initial emotion feature extractor according to the target loss data to obtain a target emotion feature extractor.

[0095] In step 201, the speech sample signal is used to represent the speech uttered by a certain speaker. The speech sample signal is different from the target speech signal described above in that the speech sample signal is used in the model training phase, and the target speech signal is used in the model use phase. The EMO-DB can be selected as the speech emotion dataset, which contains 7 emotion categories: anger, happiness, sadness, neutral, disgust, fear and surprise. The dataset is divided into a training set, a validation set and a test set in a ratio of 7:2:1. Ensure that the emotion labels in the dataset match the corresponding speech sample signals. At the same time, data augmentation is performed on the dataset by adding noise, changing volume, adjusting speech speed, etc. to increase the diversity and robustness of the data. The way to obtain the sample spectrogram feature vector is basically similar to the way to obtain the target spectrogram feature vector described above, and will not be described again.

[0096] In step 202, the initial emotion feature extractor is a neural network structure used to extract emotion-related features from the input as output. The sample spectrogram feature vector is input into the initial emotion feature extractor, and the sample spectrogram feature vector is subjected to emotion feature transformation by the initial emotion feature extractor, i.e. emotion feature mapping, to obtain a sample emotion latent feature vector.

[0097] In step 203, considering that the sample emotion latent feature vector not only contains feature information with a relatively high degree of relevance to emotion classification, but also contains feature information with a relatively low degree of relevance to emotion classification, the sample emotion latent feature vector is subjected to feature sampling to obtain at least two sample emotion sampling feature vectors. When performing feature sampling, a pre-trained feature sampler can be specifically used to perform feature sampling on the sample emotion latent feature vector. The feature sampler is a neural network structure. The obtained sample emotion sampling feature vectors can constitute a large part of the sample emotion latent feature vector, which is equivalent to filtering out some feature information with a relatively low degree of relevance to emotion classification by using feature sampling. Any two sample emotion sampling feature vectors do not overlap. The dimensions of any two sample emotion sampling feature vectors can be the same or different.

[0098] In step 204, a pre-trained decoder can be used to decode each emotion sampling feature vector to obtain a speech reconstruction feature vector. The decoder can consist of a fully connected layer. The speech reconstruction feature vector is equivalent to the reconstructed speech signal. The dimension of the speech reconstruction feature vector is the same as that of the sample spectrogram feature vector. Since there can be noise, distortion, pitch change, etc. in the speech sample signal, the sample emotion category identified from the speech sample signal can be inaccurate or unreliable. This embodiment can compare the difference between the reconstructed speech signal and the speech sample signal to determine whether the initial emotion feature extractor can effectively capture the emotion information in the speech, thereby improving the credibility and reliability of the initial emotion feature extractor.

[0099] In step 205, loss reconstruction is performed according to the voice sample signal and the voice reconstruction feature vector, to obtain target loss data. The target loss data can indicate the difference between the reconstructed voice signal and the voice sample signal. In the loss reconstruction, a similarity function such as cosine similarity, Euclidean distance, etc. can be used.

[0100] In step 206, the initial emotion feature extractor is parameter adjusted according to the target loss data, to obtain a target emotion feature extractor. Specifically, in the parameter adjustment process, the target loss data is compared with a preset first loss threshold. If the target loss data is greater than the first loss threshold, the parameter adjustment of the initial emotion feature extractor is continued until the target loss data is less than or equal to the first loss threshold, and then the parameter adjustment is stopped to obtain the target emotion feature extractor. The Adam optimizer can be used for parameter updating, with a learning rate of 0.001, a batch size of 32, and a training round number of 50. The target emotion feature extractor can be used for emotion feature transformation on the target voice spectrum feature vector to obtain a target emotion latent feature vector.

[0101] The embodiments of steps 201 to 206 have the benefit that the sample emotion latent feature vector can be identified based on the voice sample signal, and the voice reconstruction feature vector can be reconstructed based on the sample emotion latent feature vector, so that the target emotion feature extractor can be obtained by parameter adjustment of the initial emotion feature extractor based on the difference between the voice reconstruction feature vector and the voice sample signal, and the emotion feature transformation performance of the target emotion feature extractor can be improved.

[0102] In an embodiment, referring to Figure 3 Before step 206, the voice emotion recognition method of the present embodiment can further include updating the target loss data, specifically including:

[0103] In step 301, feature selection is performed on each sample emotion sampling feature vector to obtain a sample key emotion feature vector.

[0104] In step 302, the sample key emotion feature vector is classified by a preset initial classifier to obtain a sample emotion category.

[0105] In step 303, loss calculation is performed according to the sample emotion category and a preset first emotion category label to obtain first category loss data.

[0106] In step 304, the target loss data is updated according to the first category loss data.

[0107] In step 301, the purpose of feature selection is to determine a feature vector with high relevance to emotion classification from each sample emotion sampling feature vector, that is, to obtain a sample key emotion feature vector. Specifically, each sample emotion sampling feature vector can be input into a feature selector for feature selection. The feature selector is constructed by a self-attention mechanism.

[0108] In an embodiment, step 301 can include: performing relevance calculation on each sample emotion sampling feature vector and other sample emotion sampling feature vectors to obtain sample emotion feature relevance; determining the weight of each target emotion sampling feature vector according to the sample emotion feature relevance to obtain a sample weight; and extracting a sample key emotion feature vector from each sample emotion sampling feature vector according to the sample weight.

[0109] Specifically, the relevance calculation is performed on each sample emotion sampling feature vector and other sample emotion sampling feature vectors by using a self-attention layer to obtain the sample emotion feature relevance corresponding to each sample emotion sampling feature vector. The sample emotion feature relevance can be mapped to a weight by using an activation function to obtain the sample weight corresponding to each sample emotion sampling feature vector. For example, the activation function is a softmax function. The higher the sample weight is, the higher the relevance of the sample emotion sampling feature vector to emotion classification is. Specifically, each sample emotion sampling feature vector can be arranged in descending order according to the sample weight to obtain a sample arrangement position sequence; and the sample emotion sampling feature vector with a sample arrangement position sequence smaller than a preset position sequence threshold is determined as the sample key emotion feature vector. Alternatively, the sample emotion sampling feature vector corresponding to the maximum weight can be selected as the sample key emotion feature vector.

[0110] The above embodiment has the benefit of being able to determine a sample key emotion feature vector with high universality.

[0111] In step 302, the purpose of performing emotion classification by using an initial classifier is to predict the emotion category to which the speech sample signal belongs from the sample key emotion feature vector, that is, to obtain a sample emotion category. The initial classifier can be composed of a fully connected layer. The sample emotion category can be a single category, and the sample emotion category can include at least one of the following: anger, happiness, sadness, neutrality, disgust, fear, and surprise.

[0112] In step 303, the first emotion category label is a single label, and the first emotion category label indicates the emotion category to which the speech sample signal belongs. The first emotion category label can include at least one of the following: anger, happiness, sadness, neutrality, disgust, fear, and surprise. The loss calculation can be performed according to the sample emotion category and the first emotion category label to obtain first category loss data. The first category loss data can indicate the difference between the sample emotion category and the first emotion category label. The cross-entropy loss function can be used when performing the loss calculation.

[0113] In step 304, the target loss data is updated according to the first category loss data. For example, the first category loss data and the target loss data are weighted and summed to obtain the updated target loss data.

[0114] The embodiment of steps 301 to 304 has the benefit that by updating the target loss data, the influence of the difference between the sample emotion category and the first emotion category label on the target loss data is embodied, and the accuracy of the target loss data is further improved.

[0115] In step 103 of some embodiments, the target emotion feature extractor is pre-trained, and the target emotion feature extractor is a neural network structure for extracting emotion-related features as output from the input. The target emotion feature extractor has the same structure but different parameters as the initial emotion feature extractor.

[0116] The target emotion feature extractor includes a target convolutional layer and a gated recurrent unit network layer. In an embodiment, referring to Figure 4 , step 103 can include:

[0117] Step 401, performing convolutional processing on the target spectrogram feature vector through the target convolutional layer to obtain an emotion initial feature vector;

[0118] Step 402, performing context feature extraction on the emotion initial feature vector through the gated recurrent unit network layer to obtain a target emotion latent feature vector.

[0119] In step 401, the target convolutional layer includes a first convolutional layer, a second convolutional layer, and a flattening layer. The size of the convolution kernel of the first convolutional layer is smaller than the size of the convolution kernel of the second convolutional layer. In an embodiment, referring to Figure 5 , step 401 can include:

[0120] Step 501, performing first convolution on the target spectrogram feature vector through the first convolutional layer to obtain a first emotion initial feature vector;

[0121] Step 502, performing second convolution on the first emotion initial feature vector through the second convolutional layer to obtain a second emotion initial feature vector;

[0122] Step 503, performing one-dimensional vectorization on the second emotion initial feature vector through the flattening layer to obtain an emotion initial feature vector.

[0123] In an example, the target convolutional layer can include 1 first convolutional layer, 2 second convolutional layers, and a flatten layer. The first convolutional layer has a kernel size of 5*5. The second convolutional layer has a kernel size of 3*3. The flatten layer is also referred to as a flatten layer.

[0124] The embodiment of the above steps 501 to 503 has the benefit that the first convolution with a large kernel size and the second convolution with a small kernel size in combination with the one-dimension of the vector ensure that the obtained emotion initial feature vector contains emotion category related information, i.e., the convolution processing accuracy is improved.

[0125] In step 402, the gated recurrent unit network layer is composed of at least two gated recurrent units. The gated recurrent unit is GRU (Gated Recurrent Unit). The number of gated recurrent units depends on the real-time requirement, such as being set to 5 to 10.

[0126] The embodiment of the above steps 401 to 402 has the benefit that the synergistic effect of the target convolutional layer and the gated recurrent unit network layer can fully understand the timing information and context information in the target speech signal, and can extract the target emotion latent feature vector from the target spectrum feature vector, with high accuracy.

[0127] In step 104 of some embodiments, the target emotion latent feature vector is feature sampled to obtain at least two target emotion sampling feature vectors.

[0128] Considering that the target emotion latent feature vector contains not only feature information with a relatively high degree of relevance to emotion classification, but also feature information with a relatively low degree of relevance to emotion classification, the target emotion latent feature vector is feature sampled to obtain at least two target emotion sampling feature vectors. When feature sampling is performed, a feature sampler can be pre-trained to perform feature sampling on the target emotion latent feature vector. The feature sampler is a neural network structure. The obtained target emotion sampling feature vectors can constitute a large part of the target emotion latent feature vector, which is equivalent to filtering out some feature information with a relatively low degree of relevance to emotion classification by using feature sampling. Any two target emotion sampling feature vectors do not overlap. The dimensions of any two target emotion sampling feature vectors can be the same or different.

[0129] In step 105 of some embodiments, feature selection is performed on each target emotion sample feature vector to obtain a target key emotion feature vector. The purpose of performing feature selection is to determine a feature vector with high correlation with emotion classification from each target emotion sample feature vector, i.e., to obtain a target key emotion feature vector. Specifically, each target emotion sample feature vector can be input into a feature selector for feature selection. The feature selector is constructed by using a self-attention mechanism. The self-attention mechanism is used to extract key emotion features, thereby improving the accuracy and robustness of speech emotion recognition. In addition, the attention mechanism can be used to explain the attention and discrimination of the target emotion feature extractor for different parts of the speech signal, thereby improving the explainability and visualization of the target emotion feature extractor.

[0130] In an embodiment, with reference to Figure 6 , step 301 can include:

[0131] Step 601, correlation calculation is performed on each target emotion sample feature vector and other target emotion sample feature vectors to obtain a target emotion feature correlation;

[0132] Step 602, according to the target emotion feature correlation, the weight of each target emotion sample feature vector is determined to obtain a target weight;

[0133] Step 603, according to the target weight, a target key emotion feature vector is extracted from each target emotion sample feature vector.

[0134] In step 601, a self-attention layer is used to perform correlation calculation on each target emotion sample feature vector and other target emotion sample feature vectors to obtain a target emotion feature correlation corresponding to each target emotion sample feature vector.

[0135] In step 602, an activation function can be used to map the weights of the target emotion feature correlation to obtain a target weight corresponding to each target emotion sample feature vector. For example, the activation function is a softmax function.

[0136] In step 603, the higher the target weight, the higher the correlation between the target emotion sample feature vector and the emotion classification. Specifically, each target emotion sample feature vector can be arranged in descending order according to the target weight to obtain a target arrangement position sequence; a target emotion sample feature vector with a target arrangement position sequence smaller than a preset position sequence threshold is determined as a target key emotion feature vector. Alternatively, a target emotion sample feature vector corresponding to the maximum target weight can be selected as a target key emotion feature vector.

[0137] The embodiments of steps 601 to 603 have the advantage of being able to determine a target key emotion feature vector with high universality.

[0138] In an embodiment, referring to Figure 7 Before step 106, the voice emotion recognition method provided by the embodiment can further include: training the target classifier, specifically including:

[0139] Step 701: loss calculation is performed according to the sample emotion category and the preset second emotion category label, to obtain second category loss data.

[0140] Step 702: parameter adjustment is performed on the initial classifier according to the second category loss data, to obtain the target classifier.

[0141] In step 701, the second emotion category label is a multi-label. The second emotion category label indicates the emotion category to which the voice sample signal belongs and the emotion degree of the emotion category. For example, the second emotion category label can include at least one of the following: anger 1, anger 2, anger 3, happiness 1, happiness 2, happiness 3, sadness 1, sadness 2, sadness 3, neutrality 1, neutrality 2, neutrality 3, disgust 1, disgust 2, disgust 3, fear 1, fear 2, fear 3, surprise 1, surprise 2, and surprise 3. Among them, 1, 2, and 3 respectively represent low, medium, and high emotion degrees. The loss calculation can be performed according to the emotion category and the second emotion category label, to obtain the second category loss data. The second category loss data can indicate the difference between the sample emotion category and the second emotion category label. The cross-entropy loss function can be used for loss calculation.

[0142] In step 702, in the parameter adjustment process, the second category loss data is compared with a preset second loss threshold. If the second category loss data is greater than the second loss threshold, the parameter adjustment on the initial classifier is continued until the second category loss data is less than or equal to the second loss threshold, and then the parameter adjustment is stopped to obtain the target classifier. The Adam optimizer can be used for parameter adjustment, the learning rate is set to 0.001, the batch size is set to 32, and the number of training rounds is set to 50. The target classifier can be used for emotion classification on the target key emotion feature vector appearing below to obtain a target emotion category.

[0143] The embodiment of steps 701 to 702 has the benefit that the initial classifier can be adjusted in parameters based on the difference between the sample emotion category and the second emotion category label to obtain the target classifier, and the emotion classification granularity of the target classifier can be improved.

[0144] In step 106, the target key emotion feature vector is classified by a pre-trained target classifier to obtain a target emotion category. The target classifier can be composed of a full connection layer. If the initial classifier is directly used as the target classifier, the target classifier has the same structure and parameters as the initial classifier, and the target emotion category can include at least one of the following: anger, happiness, sadness, neutrality, disgust, fear, and surprise. If the target classifier is obtained by training the initial classifier based on the second category loss data, the target classifier has the same structure but different parameters from the initial classifier, and the target emotion category label can include at least one of the following: anger 1, anger 2, anger 3, happiness 1, happiness 2, happiness 3, sadness 1, sadness 2, sadness 3, neutrality 1, neutrality 2, neutrality 3, disgust 1, disgust 2, disgust 3, fear 1, fear 2, fear 3, surprise 1, surprise 2, and surprise 3.

[0145] Referring to Figure 8 The embodiments of the present application also provide a voice emotion recognition device, which can implement the voice emotion recognition method described above. Figure 8 A module structure block diagram of the voice emotion recognition device provided by the embodiments of the present application is shown in the figure. The device includes a voice acquisition module 801, a feature extraction module 802, a feature transformation module 803, a feature sampling module 804, a feature selection module 805, and an emotion classification module 806. The voice acquisition module 801 is configured to acquire a target voice signal. The feature extraction module 802 is configured to perform acoustic spectrum feature extraction on the target voice signal to obtain a target acoustic spectrum feature vector. The feature transformation module 803 is configured to perform emotion feature transformation on the target acoustic spectrum feature vector by using a pre-trained target emotion feature extractor to obtain a target emotion latent feature vector. The feature sampling module 804 is configured to perform feature sampling on the target emotion latent feature vector to obtain at least two target emotion sampling feature vectors. The feature selection module 805 is configured to perform feature selection on each target emotion sampling feature vector to obtain a target key emotion feature vector. The emotion classification module 806 is configured to perform emotion classification on the target key emotion feature vector by using a pre-trained target classifier to obtain a target emotion category.

[0146] In an embodiment, the voice emotion recognition device further includes a first training module configured to train the target emotion feature extractor.

[0147] In another embodiment, the voice emotion recognition device further includes a first training module configured to update the target loss data.

[0148] In an embodiment, the voice emotion recognition device further includes a second training module configured to train the target classifier.

[0149] It should be noted that the specific implementation of the voice emotion recognition device is basically the same as the specific embodiment of the voice emotion recognition method described above, and will not be repeated here.

[0150] The embodiments of the present application further provide an electronic device, which comprises a memory, a processor, a program stored in the memory and capable of running on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the voice emotion recognition method. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0151] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, which comprises:

[0152] The processor 901 can be implemented in the form of a general CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to realize the technical solutions provided by the embodiments of the present application;

[0153] The memory 902 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 902 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are saved in the memory 902 and are called and executed by the processor 901 to realize the voice emotion recognition method of the embodiments of the present application;

[0154] The input / output interface 903 is used to realize information input and output;

[0155] The communication interface 904 is used to realize the communication interaction between the device and other devices, which can realize communication through a wired manner (such as a USB, a network cable, etc.) or a wireless manner (such as a mobile network, a WI-FI, a Bluetooth, etc.);

[0156] The bus 905 transmits information between various components (such as the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device;

[0157] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are communicatively connected with each other through a bus 905.

[0158] The embodiment of the present application further provides a storage medium, which is a computer readable storage medium, used for computer readable storage, and stores one or more programs, which can be executed by one or more processors to implement the voice emotion recognition method.

[0159] The memory is a non-transitory computer readable storage medium, and can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can further include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0160] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0161] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0162] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiments of the present application.

[0163] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0164] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a given step or its integral presence in the process, method, system, article, or apparatus having been made with a wider scope. The use of notation such as "first", "second", "third", etc. does not generally limit the areas, but is used to connect like elements or to distinguish one claim from another. These terms can be used interchangeably when appropriate. Terms concerning the relative position of elements can be interpreted such that their use adheres to their normal meaning, but they can also be interpreted to mean the opposite according to specific claims.

[0165] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are only A, only B, and A and B at the same time. Where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and back associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c, can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0166] In several embodiments provided by the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the above-described device embodiments are only illustrative, for example, the division of the units is only a logical functional division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0167] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the application.

[0168] In addition, each of the functional units in the embodiments of the present application can be integrated in one processing unit, or each unit can exist alone physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0169] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially, or the part that contributes to the prior art, or all or a part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0170] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the embodiments of the present application. Any modification, equivalent replacement, and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A voice emotion recognition method, characterized in that, The method includes: Acquire the target speech signal; The target speech signal is subjected to spectral feature extraction to obtain the target spectral feature vector; Obtain the spectral features of the speech sample signal to obtain the sample spectral feature vector; The sample acoustic spectrum feature vector is transformed by a preset initial emotion feature extractor to obtain the sample emotion potential feature vector. The potential feature vectors of the sample emotions are sampled to obtain at least two sample emotion sampling feature vectors. Each of the emotion sampling feature vectors is decoded to obtain a speech reconstruction feature vector; wherein the dimension of the speech reconstruction feature vector is the same as the dimension of the sample spectral feature vector. Loss reconstruction is performed based on the speech sample signal and the speech reconstruction feature vector to obtain the target loss data; The parameters of the initial emotion feature extractor are adjusted based on the target loss data to obtain the target emotion feature extractor. The target emotion feature vector is transformed by the target spectral feature extractor to obtain the target emotion potential feature vector. Feature sampling is performed on the potential feature vector of the target emotion to obtain at least two sampled feature vectors of the target emotion; The correlation between each target emotion sampling feature vector and other target emotion sampling feature vectors is calculated to obtain the target emotion feature correlation. Based on the correlation of the target emotion features, the weights of each of the target emotion sampling feature vectors are determined to obtain the target weights; Based on the target weights, extract the target key emotion feature vectors from each of the target emotion sampling feature vectors; The target key emotion feature vector is classified into emotion categories by using a pre-trained target classifier.

2. The method according to claim 1, characterized in that, Before adjusting the parameters of the initial emotion feature extractor based on the target loss data to obtain the target emotion feature extractor, the method further includes: Updating the target loss data specifically includes: Feature selection is performed on the emotion sampling feature vectors of each sample to obtain the key emotion feature vectors of the samples. The sample's key emotional feature vectors are classified using a preset initial classifier to obtain the sample's emotional category. Loss calculation is performed based on the sample emotion category and the preset first emotion category label to obtain first category loss data; wherein, the first emotion category label is a single label, and the first emotion category label indicates the emotion category to which the speech sample signal belongs; The target loss data is updated based on the first category of loss data.

3. The method according to claim 2, characterized in that, Before classifying the target key emotion feature vector using a pre-trained target classifier to obtain the target emotion category, the method further includes: The target classifier is trained and includes: Loss calculation is performed based on the sample emotion category and the preset second emotion category label to obtain second category loss data; wherein, the second emotion category label is a multi-label, and the second emotion category label indicates the emotion category to which the speech sample signal belongs and the emotion level of the emotion category; The initial classifier is adjusted based on the second category loss data to obtain the target classifier.

4. The method according to any one of claims 1 to 3, characterized in that, The target emotion feature extractor includes a target convolutional layer and a gated recurrent unit network layer; The step of performing emotional feature transformation on the target spectral feature vector using a pre-trained target emotion feature extractor to obtain the target emotion latent feature vector includes: The target acoustic spectrum feature vector is convolved by the target convolutional layer to obtain the initial emotion feature vector. The initial emotion feature vector is extracted using the gated recurrent unit network layer to obtain the target emotion potential feature vector.

5. The method according to claim 4, characterized in that, The target convolutional layer includes a first convolutional layer, a second convolutional layer, and a flattening layer, wherein the kernel size of the first convolutional layer is smaller than the kernel size of the second convolutional layer; The step of convolving the target acoustic spectrum feature vector through the target convolutional layer to obtain the initial emotion feature vector includes: The first convolution is performed on the target acoustic spectrum feature vector by the first convolutional layer to obtain the first emotion initial feature vector; The first initial feature vector of emotion is convolved by the second convolutional layer to obtain the second initial feature vector of emotion; The second initial emotion feature vector is vectorized into one dimension by the flattening layer to obtain the initial emotion feature vector.

6. A voice emotion recognition device, characterized in that, The device includes: The voice acquisition module is used to acquire the target voice signal; The feature extraction module is used to extract the acoustic spectrum features from the target speech signal to obtain the target acoustic spectrum feature vector; The first training module is used for: acquiring the spectroscopic features of a speech sample signal to obtain a sample spectroscopic feature vector; performing feature transformation on the sample spectroscopic feature vector using a preset initial emotion feature extractor to obtain a sample emotion latent feature vector; sampling the sample emotion latent feature vector to obtain at least two sample emotion sampling feature vectors; decoding each of the emotion sampling feature vectors to obtain a speech reconstruction feature vector; wherein the dimension of the speech reconstruction feature vector is the same as the dimension of the sample spectroscopic feature vector; performing loss reconstruction based on the speech sample signal and the speech reconstruction feature vector to obtain target loss data; and adjusting the parameters of the initial emotion feature extractor based on the target loss data to obtain a target emotion feature extractor. The feature transformation module is used to perform emotional feature transformation on the target acoustic spectrum feature vector through the target emotion feature extractor to obtain the target emotion potential feature vector. The feature sampling module is used to sample the potential feature vector of the target emotion to obtain at least two target emotion sampling feature vectors. The feature selection module is used to: calculate the correlation between each target emotion sampling feature vector and other target emotion sampling feature vectors to obtain target emotion feature correlation; determine the weight of each target emotion sampling feature vector based on the target emotion feature correlation to obtain target weight; and extract target key emotion feature vectors from each target emotion sampling feature vector based on the target weight. The emotion classification module is used to classify the target key emotion feature vector using a pre-trained target classifier to obtain the target emotion category.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images