Voice emotion recognition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a voice emotion recognition method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence and the field of financial science and technology. The method comprises the following steps: acquiring a target voice signal; performing sound spectrum feature extraction on the target voice signal to obtain a target sound spectrum feature vector; performing emotional feature transformation on the target sound spectrum feature vector through a pre-trained target emotional feature extractor to obtain a target emotional potential feature vector; performing feature sampling on the target emotion potential feature vector to obtain at least two target emotion sampling feature vectors; performing feature selection on each target emotion sampling feature vector to obtain a target key emotion feature vector; and performing emotion classification on the target key emotion feature vector through a pre-trained target classifier to obtain a target emotion category. The embodiment of the invention can improve the emotion recognition accuracy and improve the recognition efficiency.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence technology and financial technology, and in particular to a speech emotion recognition method and device, electronic equipment, and storage medium. Background Art

[0002] Speech emotion recognition is a technology that analyzes and identifies the speaker's emotional state from speech signals. The existing technology proposes a method based on deep learning, which requires directly learning emotional features from speech signals and performing emotion classification. However, it is difficult to directly learn emotional features from speech signals, usually requiring a large amount of computing resources, and there is still much room for improvement in accuracy.

[0003] Therefore, how to provide a speech emotion recognition method that can improve the accuracy of emotion recognition and improve recognition efficiency has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a speech emotion recognition method and device, an electronic device, and a storage medium, which can improve the accuracy of emotion recognition and improve the recognition efficiency.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a method for speech emotion recognition, the method comprising:

[0006] Acquire a target speech signal;

[0007] Extracting the spectral features of the target speech signal to obtain a target spectral feature vector;

[0008] Performing an emotional feature transformation on the target sound spectrum feature vector by using a pre-trained target emotion feature extractor to obtain a target emotion latent feature vector;

[0009] Performing feature sampling on the target emotion potential feature vector to obtain at least two target emotion sampling feature vectors;

[0010] Performing feature selection on each of the target emotion sampling feature vectors to obtain a target key emotion feature vector;

[0011] The target key emotion feature vector is subjected to emotion classification by a pre-trained target classifier to obtain a target emotion category.

[0012] In some embodiments, before the pre-trained target emotion feature extractor performs feature transformation on the target sound spectrum feature vector to obtain the target emotion potential feature vector, the method further includes:

[0013] The target emotion feature extractor is obtained by training, specifically comprising:

[0014] Acquire the spectral features of the speech sample signal to obtain a sample spectral feature vector;

[0015] Performing feature transformation on the sample sound spectrum feature vector by a preset initial emotion feature extractor to obtain a sample emotion potential feature vector;

[0016] Sampling the sample emotion potential feature vector to obtain at least two sample emotion sampling feature vectors;

[0017] Decoding each of the emotion sampling feature vectors to obtain a speech reconstruction feature vector; wherein the dimension of the speech reconstruction feature vector is the same as the dimension of the sample spectrum feature vector;

[0018] Perform loss reconstruction according to the speech sample signal and the speech reconstruction feature vector to obtain target loss data;

[0019] The parameters of the initial emotion feature extractor are adjusted according to the target loss data to obtain the target emotion feature extractor.

[0020] In some embodiments, before adjusting the parameters of the initial emotion feature extractor according to the target loss data to obtain the target emotion feature extractor, the method further includes:

[0021] Updating the target loss data, specifically including:

[0022] Performing feature selection on each of the sample emotion sampling feature vectors to obtain a sample key emotion feature vector;

[0023] Performing emotion classification on the sample key emotion feature vector by using a preset initial classifier to obtain a sample emotion category;

[0024] Performing loss calculation according to the sample emotion category and a preset first emotion category label to obtain first category loss data; wherein the first emotion category label is a single label, and the first emotion category label indicates the emotion category to which the speech sample signal belongs;

[0025] The target loss data is updated according to the first category loss data.

[0026] In some embodiments, before performing emotion classification on the target key emotion feature vector by a pre-trained target classifier to obtain a target emotion category, the method further includes:

[0027] The target classifier is obtained by training, specifically including:

[0028] Performing loss calculation according to the sample emotion category and a preset second emotion category label to obtain second category loss data; wherein the second emotion category label is a multi-label, and the second emotion category label indicates the emotion category to which the speech sample signal belongs and the emotion degree of the emotion category;

[0029] The parameters of the initial classifier are adjusted according to the second category loss data to obtain the target classifier.

[0030] In some embodiments, the target emotion feature extractor includes a target convolutional layer and a gated recurrent unit network layer;

[0031] The target emotion feature extractor is used to perform emotion feature transformation on the target sound spectrum feature vector to obtain the target emotion potential feature vector, including:

[0032] Performing convolution processing on the target sound spectrum feature vector through the target convolution layer to obtain an initial emotion feature vector;

[0033] The context feature extraction is performed on the initial emotion feature vector through the gated recurrent unit network layer to obtain the target emotion potential feature vector.

[0034] In some embodiments, the target convolution layer includes a first convolution layer, a second convolution layer and a flattening layer, and the convolution kernel size of the first convolution layer is smaller than the convolution kernel size of the second convolution layer;

[0035] The step of performing convolution processing on the target sound spectrum feature vector through the target convolution layer to obtain an initial emotion feature vector includes:

[0036] Performing a first convolution on the target sound spectrum feature vector through the first convolution layer to obtain a first emotion initial feature vector;

[0037] Performing a second convolution on the first initial emotion feature vector through the second convolution layer to obtain a second initial emotion feature vector;

[0038] The second initial emotion feature vector is transformed into a one-dimensional vector through the flattening layer to obtain the initial emotion feature vector.

[0039] In some embodiments, performing feature selection on each of the target emotion sampling feature vectors to obtain a target key emotion feature vector includes:

[0040] Performing correlation calculation on each of the target emotion sampling feature vectors and other target emotion sampling feature vectors to obtain target emotion feature correlation;

[0041] Determine the weight of each of the target emotion sampling feature vectors according to the target emotion feature correlation to obtain a target weight;

[0042] The target key emotion feature vector is extracted from each of the target emotion sampling feature vectors according to the target weight.

[0043] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech emotion recognition device, the device comprising:

[0044] A speech acquisition module, used to acquire a target speech signal;

[0045] A feature extraction module is used to extract the spectral features of the target speech signal to obtain a target spectral feature vector;

[0046] A feature transformation module, used for performing an emotion feature transformation on the target sound spectrum feature vector by using a pre-trained target emotion feature extractor to obtain a target emotion potential feature vector;

[0047] A feature sampling module, used for performing feature sampling on the target emotion potential feature vector to obtain at least two target emotion sampling feature vectors;

[0048] A feature selection module, used to perform feature selection on each of the target emotion sampling feature vectors to obtain a target key emotion feature vector;

[0049] The emotion classification module is used to perform emotion classification on the target key emotion feature vector through a pre-trained target classifier to obtain a target emotion category.

[0050] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the speech emotion recognition method described in the first aspect when executing the computer program.

[0051] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the speech emotion recognition method described in the first aspect above.

[0052] The speech emotion recognition method, speech emotion recognition device, electronic device and storage medium proposed in the present application, when the target emotion category needs to be identified from the target speech signal, the method first extracts the spectral feature of the target speech signal to obtain the target spectral feature vector, and then performs feature transformation on the target spectral feature vector to obtain the target emotion potential feature vector. In this way, instead of directly extracting the feature vector related to emotion from the target speech signal, the method first extracts the spectral feature and then performs the emotion feature transformation, so that the feature vector related to emotion can be extracted from the target speech signal, and the extraction difficulty is low. Considering that the target emotion potential feature vector contains feature information with different degrees of influence on emotion classification, feature sampling is first used to obtain at least two target emotion sampling feature vectors, and then feature selection is used to obtain the target key emotion feature vector. In this way, the feature vector with a high degree of influence on emotion classification can be extracted from the target speech signal. Then, the target classifier is used to perform emotion classification to obtain the target emotion category. In summary, when extracting features for emotion classification from the target speech signal, this embodiment reduces the difficulty of feature extraction, can save computing resources, and also ensures feature accuracy. In summary, this embodiment can improve the accuracy of emotion recognition and improve recognition efficiency.

[0053] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a method for speech emotion recognition provided by an embodiment of the present application;

[0055] Figure 2 is a flow chart of a speech emotion recognition method provided by another embodiment of the present application;

[0056] Figure 3 is a flow chart of a speech emotion recognition method provided by another embodiment of the present application;

[0057] Figure 4 yes Figure 1 Flowchart of step 103 in

[0058] Figure 5 yes Figure 4 Flow chart of step 401 in;

[0059] Figure 6 yes Figure 1 Flowchart of step 105 in;

[0060] Figure 7 is a flow chart of a speech emotion recognition method provided by another embodiment of the present application;

[0061] Figure 8 It is a module structure block diagram of the speech emotion recognition device provided in an embodiment of the present application;

[0062] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] First, some nouns involved in this application are analyzed:

[0067] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0069] Speech emotion recognition technology: is a technology that analyzes and identifies the speaker's emotional state from speech signals. It can be used to improve the naturalness and friendliness of human-computer interaction, as well as to provide more emotional information and services. The main task of speech emotion recognition is to convert speech signals into corresponding emotion categories, such as anger, happiness, sadness, neutrality, etc.

[0070] In the field of financial technology, voice emotion recognition technology has many application scenarios, such as: (1) Customer emotion analysis: Financial institutions can use voice emotion recognition technology to analyze the emotions of customers during phone calls. By monitoring and analyzing customers' voice emotions in real time, financial institutions can better understand customers' attitudes and emotions and take corresponding countermeasures based on the recognition results, such as adjusting customer service strategies or guiding customers' investment directions. (2) Voice assistant and emotional interaction: Embedding voice assistant and emotional interaction functions in financial services can provide users with personalized service experience. Voice assistants can analyze users' voice emotions, understand users' emotional needs, and provide more considerate financial advice and services. (3) Anti-fraud detection: Voice emotion recognition can be used to detect anti-fraud behavior. By analyzing voice emotions in telephone transactions or customer service conversations, financial institutions can discover abnormal emotional patterns, fearful or nervous voice characteristics, and thus improve the recognition of anti-fraud behavior.

[0071] However, it is difficult for related technologies to use neural network models to directly learn emotional features from speech signals, which usually requires a lot of computing resources and there is still much room for improvement in accuracy.

[0072] Based on this, the embodiments of the present application propose a speech emotion recognition method and a speech emotion recognition device, an electronic device, and a storage medium, which can improve the accuracy of emotion recognition and improve the recognition efficiency.

[0073] The speech emotion recognition method provided in the embodiment of the present application can be applied to the terminal and the server side, and can also be software running on the server side. The server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech emotion recognition method, etc., but is not limited to the above forms.

[0074] The present application can be used in many general or special computer system environments or configurations. For example: server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0075] The embodiments of the present application provide a speech emotion recognition method and a speech emotion recognition device, an electronic device, and a storage medium, which are specifically illustrated by the following embodiments. First, the speech emotion recognition method in the embodiments of the present application is described.

[0076] It should be noted that in each specific implementation method of the present application, when it comes to performing relevant processing based on the user's voice data or other data related to the user's identity or characteristics, the user's permission or consent must be obtained first, and the collection, use and processing of such data will comply with relevant laws, regulations and standards.

[0077] Reference Figure 1 , Figure 1 It is an optional flowchart of the speech emotion recognition method provided in an embodiment of the present application, which may include but is not limited to steps 101 to 106.

[0078] Step 101, obtaining a target speech signal;

[0079] Step 102, extracting the spectral features of the target speech signal to obtain a target spectral feature vector;

[0080] Step 103, performing an emotion feature transformation on the target sound spectrum feature vector by a pre-trained target emotion feature extractor to obtain a target emotion latent feature vector;

[0081] Step 104, performing feature sampling on the target emotion potential feature vector to obtain at least two target emotion sampling feature vectors;

[0082] Step 105, performing feature selection on each target emotion sampling feature vector to obtain a target key emotion feature vector;

[0083] Step 106 , performing emotion classification on the target key emotion feature vector by using a pre-trained target classifier to obtain a target emotion category.

[0084] In the steps 101 to 106 shown in the embodiment of the present application, when it is necessary to identify the target emotion category from the target speech signal, the target speech signal is first subjected to spectral feature extraction to obtain the target spectral feature vector, and then the target spectral feature vector is subjected to feature transformation to obtain the target emotion potential feature vector. In this way, instead of directly extracting the target emotion potential feature vector from the target speech signal, the spectral feature extraction is first performed and then the emotion feature transformation is performed, so that the feature vector related to the emotion can be extracted from the target speech signal, and the extraction difficulty is low. Considering that the target emotion potential feature vector contains feature information with different degrees of influence on emotion classification, feature sampling is first used to obtain at least two target emotion sampling feature vectors, and then feature selection is used to obtain the target key emotion feature vector. In this way, the feature information with a high degree of influence on emotion classification is extracted from the target speech signal. Then, the target classifier is used to perform emotion classification to obtain the target emotion category. In summary, when extracting features for emotion classification from the target speech signal, this embodiment reduces the difficulty of feature extraction, can save computing resources, and also ensures feature accuracy. In summary, this embodiment can improve the accuracy of emotion recognition and improve recognition efficiency.

[0085] In step 101 of some embodiments, a target voice signal is obtained. The target voice signal is used to represent the voice uttered by a certain speaker. Specifically, when the voice emotion recognition method is applied to a terminal, the target voice signal can be obtained by recording, Bluetooth transmission, wired transmission, or downloading. When the target voice signal is obtained by recording, the terminal is correspondingly configured with a microphone, and audio is collected through the microphone to achieve recording of the target voice signal. When the voice emotion recognition method is applied to a server, the voice sample data can be uploaded to the server by the terminal, or downloaded by the server from other servers or databases.

[0086] In step 102 of some embodiments, the target speech signal is subjected to spectral feature extraction to obtain a target spectral feature vector. Spectral features refer to the distribution characteristics of sound in the frequency domain, which are usually displayed by a spectrogram. The spectrogram reflects the frequency components and intensity of the sound, and can help identify and analyze the characteristics of the sound, such as pitch, timbre, volume, and frequency components. For example, the spectral feature is Mel Frequency Cepstrum Coefficients (MFCC), which can reflect the human perception characteristics of sound.

[0087] The MFCC features are calculated by the following steps: first, the target speech signal is pre-emphasized and framed; then, each frame is windowed with a Hamming window and fast Fourier transformed; then, the spectrum is filtered by a Mel filter bank and logarithmically operated; finally, the filter bank output is discrete cosine transformed to obtain the MFCC coefficients. In addition, other acoustic features such as the fundamental frequency (F0) and energy (E) of the speech signal are extracted and spliced ​​together with the MFCC features to form a richer feature vector. In this embodiment, the sampling rate can be set to 16000 Hz, the frame length to 25 ms, the frame shift to 10 ms, the number of Mel filter banks to 40, and the number of MFCC coefficients to 13.

[0088] In one embodiment, referring to Figure 2 Before step 103, the speech emotion recognition method provided by this embodiment may further include: training to obtain a target emotion feature extractor, specifically including:

[0089] Step 201, acquiring the spectral features of the speech sample signal to obtain a sample spectral feature vector;

[0090] Step 202, performing feature transformation on the sample sound spectrum feature vector by a preset initial emotion feature extractor to obtain the sample emotion potential feature vector;

[0091] Step 203, sampling the sample emotion potential feature vector to obtain at least two sample emotion sampling feature vectors;

[0092] Step 204, decoding each emotion sampling feature vector to obtain a speech reconstruction feature vector;

[0093] Step 205, performing loss reconstruction according to the speech sample signal and the speech reconstruction feature vector to obtain target loss data;

[0094] Step 206, adjusting the parameters of the initial emotion feature extractor according to the target loss data to obtain a target emotion feature extractor.

[0095] In step 201, the speech sample signal is used to represent the speech uttered by a certain speaker. The difference between the speech sample signal and the above-mentioned target speech signal is that the speech sample signal is used in the model training phase, and the target speech signal is used in the model use phase. EMO-DB can be selected as the speech emotion dataset, which contains 7 emotion categories: anger, happiness, sadness, neutrality, disgust, fear and surprise. The dataset is divided into a training set, a validation set and a test set in a ratio of 7:2:1. Ensure that the emotion labels in the dataset match the corresponding speech sample signals. At the same time, the dataset is enhanced to increase the diversity and robustness of the data by adding noise, changing the volume, adjusting the speaking speed, etc. The method of obtaining the sample spectral feature vector is basically similar to the method of obtaining the target spectral feature vector mentioned above, so it will not be repeated.

[0096] In step 202, the initial emotion feature extractor is a neural network structure, which is used to extract emotion-related features from input as output. The sample sound spectrum feature vector is input into the initial emotion feature extractor, and the initial emotion feature extraction performs emotion feature transformation on the sample sound spectrum feature vector, also known as emotion feature mapping, to obtain the sample emotion potential feature vector.

[0097] In step 203, considering that the sample emotion latent feature vector contains not only feature information with a high degree of relevance to emotion classification, but also feature information with a low degree of relevance to emotion classification, feature sampling is performed on the sample emotion latent feature vector to obtain at least two sample emotion sampling feature vectors. When performing feature sampling, it can be specifically to pre-train a feature sampler to perform feature sampling on the sample emotion latent feature vector. The feature sampler is a neural network structure. The obtained individual sample emotion sampling feature vectors can constitute most of the sample emotion latent feature vector, which is equivalent to using feature sampling to filter out some feature information with a low degree of relevance to emotion classification. Any two sample emotion sampling feature vectors do not overlap. The dimensions of any two sample emotion sampling feature vectors may be the same or different.

[0098] In step 204, each emotion sampling feature vector can be decoded using a pre-trained decoder to obtain a speech reconstruction feature vector. The decoder can be composed of a fully connected layer. The speech reconstruction feature vector is equivalent to the reconstructed speech signal. The dimension of the speech reconstruction feature vector is the same as the dimension of the sample spectrum feature vector. Since the speech sample signal may contain noise, distortion, pitch change and other factors, the sample emotion category identified from the speech sample signal is inaccurate or unreliable. This embodiment can determine whether the initial emotion feature extractor can effectively capture the emotional information in the speech by comparing the difference between the reconstructed speech signal and the speech sample signal, thereby improving the credibility and reliability of the initial emotion feature extractor.

[0099] In step 205, loss reconstruction is performed according to the speech sample signal and the speech reconstruction feature vector to obtain target loss data. The target loss data can indicate the difference between the reconstructed speech signal and the speech sample signal. When performing loss reconstruction, a similarity function such as cosine similarity, Euclidean distance, etc. can be used.

[0100] In step 206, the parameters of the initial emotion feature extractor are adjusted according to the target loss data to obtain the target emotion feature extractor. Specifically, during the parameter adjustment process, the target loss data is compared with the preset first loss threshold. If the target loss data is greater than the first loss threshold, the parameters of the initial emotion feature extractor continue to be adjusted until the target loss data is less than or equal to the first loss threshold, then the parameter adjustment is stopped to obtain the target emotion feature extractor. The Adam optimizer can be used for parameter update, setting the learning rate to 0.001, the batch size to 32, and the number of training rounds to 50. The target emotion feature extractor can be used to perform an emotion feature transformation on the target spectral feature vector that appears below to obtain a target emotion potential feature vector.

[0101] The benefit of the embodiment of the above-mentioned steps 201 to 206 is that a sample emotion latent feature vector can be obtained based on speech sample signal recognition, and a speech reconstructed feature vector can be reconstructed based on the sample emotion latent feature vector, so that the parameters of the initial emotion feature extractor can be adjusted based on the difference between the speech reconstructed feature vector and the speech sample signal to obtain a target emotion feature extractor, thereby improving the emotion feature conversion performance of the target emotion feature extractor.

[0102] In one embodiment, referring to Figure 3 Before step 206, the speech emotion recognition method of this embodiment may further include: updating the target loss data, specifically including:

[0103] Step 301, performing feature selection on each sample emotion sampling feature vector to obtain a sample key emotion feature vector;

[0104] Step 302, performing emotion classification on the sample key emotion feature vector by a preset initial classifier to obtain the sample emotion category;

[0105] Step 303, performing loss calculation according to the sample emotion category and the preset first emotion category label to obtain first category loss data;

[0106] Step 304: Update target loss data according to the first category loss data.

[0107] In step 301, the purpose of feature selection is to determine the feature vector with high correlation with emotion classification from each sample emotion sampling feature vector, that is, to obtain the sample key emotion feature vector. Specifically, each sample emotion sampling feature vector can be input into a feature selector for feature selection. The feature selector is constructed by a self-attention mechanism.

[0108] In one embodiment, step 301 may include: performing correlation calculation on each sample emotion sampling feature vector and other sample emotion sampling feature vectors to obtain sample emotion feature correlation; determining the weight of each target emotion sampling feature vector based on the sample emotion feature correlation to obtain a sample weight; and extracting a sample key emotion feature vector from each sample emotion sampling feature vector based on the sample weight.

[0109] Specifically, the self-attention layer is used to calculate the correlation between each sample emotion sampling feature vector and other sample emotion sampling feature vectors to obtain the sample emotion feature correlation corresponding to each sample emotion sampling feature vector. The activation function can be used to perform weight mapping on the sample emotion feature correlation to obtain the sample weight corresponding to each sample emotion sampling feature vector. For example, the activation function is the softmax function. The higher the sample weight, the higher the correlation between the sample emotion sampling feature vector and the emotion classification. Specifically, the sample emotion sampling feature vectors can be arranged in descending order according to the sample weight to obtain the sample arrangement order; the sample emotion sampling feature vector whose sample arrangement order is less than the preset order threshold is determined as the sample key emotion feature vector. The sample emotion sampling feature vector corresponding to the largest weight can also be selected as the sample key emotion feature vector.

[0110] The benefit of the above embodiment is that it can determine the key emotion feature vector of the sample and has high universality.

[0111] In step 302, the purpose of using the initial classifier to perform emotion classification is to predict the emotion category to which the speech sample signal belongs from the sample key emotion feature vector, that is, to obtain the sample emotion category. The initial classifier can be composed of a fully connected layer. The sample emotion category can be a single category, and the sample emotion category can include at least one of the following: anger, happiness, sadness, neutral, disgust, fear and surprise.

[0112] In step 303, the first emotion category label is a single label, and the first emotion category label indicates the emotion category to which the speech sample signal belongs. The first emotion category label may include at least one of the following: anger, happiness, sadness, neutrality, disgust, fear, and surprise. Loss calculation can be performed based on the sample emotion category and the first emotion category label to obtain first category loss data. The first category loss data can indicate the difference between the sample emotion category and the first emotion category label. A cross entropy loss function can be used when performing loss calculation.

[0113] In step 304, the target loss data is updated according to the first category loss data. For example, the first category loss data and the target loss data are weighted and summed to obtain updated target loss data.

[0114] The benefit of the above-mentioned embodiment of steps 301 to 304 is that, by updating the target loss data, the influence of the difference between the sample emotion category and the first emotion category label on the target loss data is reflected, thereby further improving the accuracy of the target loss data.

[0115] In step 103 of some embodiments, the target spectrum feature vector is transformed into an emotional feature by a pre-trained target emotion feature extractor to obtain a target emotion latent feature vector. The target emotion feature extractor is similar to the initial emotion feature extractor and is also a neural network structure for extracting emotion-related features from input as output. The target emotion feature extractor has the same structure as the initial emotion feature extractor but different parameters.

[0116] The target emotion feature extractor includes a target convolution layer and a gated recurrent unit network layer. Figure 4 , step 103 may include:

[0117] Step 401, performing convolution processing on the target sound spectrum feature vector through the target convolution layer to obtain the initial emotion feature vector;

[0118] Step 402: extract context features from the initial emotion feature vector through a gated recurrent unit network layer to obtain a target emotion latent feature vector.

[0119] In step 401, the target convolution layer includes a first convolution layer, a second convolution layer, and a flattening layer. The convolution kernel size of the first convolution layer is smaller than the convolution kernel size of the second convolution layer. Figure 5 , step 401 may include:

[0120] Step 501, performing a first convolution on the target sound spectrum feature vector through a first convolution layer to obtain a first emotion initial feature vector;

[0121] Step 502, performing a second convolution on the first emotion initial feature vector through a second convolution layer to obtain a second emotion initial feature vector;

[0122] Step 503: convert the second initial emotion feature vector into a one-dimensional vector by using a flattening layer to obtain an initial emotion feature vector.

[0123] In one example, the target convolution layer may include one first convolution layer, two second convolution layers, and a flattening layer. The convolution kernel size of the first convolution layer is 5*5. The convolution kernel size of the second convolution layer is 3*3. The flattening layer is also called a flatten layer.

[0124] The benefit of the embodiment of the above steps 501 to 503 is that a large convolution kernel size is first used for the first convolution and then a small convolution kernel size is used for the second convolution and the vector is combined with one-dimensionalization, thereby ensuring that the obtained initial emotion feature vector contains information related to the emotion category, thereby improving the accuracy of the convolution processing.

[0125] In step 402, the gated recurrent unit network layer is composed of at least two gated recurrent units. The gated recurrent unit is GRU (Gated Recurrent Unit). The number of gated recurrent units depends on the real-time requirements, for example, set at 5 to 10.

[0126] The benefit of the embodiment of steps 401 to 402 is that, by utilizing the synergy of the target convolutional layer and the gated recurrent unit network layer, the timing information and context information in the target speech signal can be fully understood, and the target emotion potential feature vector can be extracted from the target spectral feature vector with high accuracy.

[0127] In step 104 of some embodiments, feature sampling is performed on the target emotion potential feature vector to obtain at least two target emotion sampled feature vectors.

[0128] Considering that the target emotion latent feature vector contains not only feature information with a high degree of relevance to emotion classification, but also feature information with a low degree of relevance to emotion classification, feature sampling is performed on the target emotion latent feature vector to obtain at least two target emotion sampling feature vectors. When performing feature sampling, it can be specifically pre-trained to perform feature sampling on the target emotion latent feature vector. The feature sampler is a neural network structure. The obtained target emotion sampling feature vectors can constitute most of the target emotion latent feature vector, which is equivalent to using feature sampling to filter out some feature information with a low degree of relevance to emotion classification. Any two target emotion sampling feature vectors do not overlap. The dimensions of any two target emotion sampling feature vectors may be the same or different.

[0129] In step 105 of some embodiments, feature selection is performed on each target emotion sampling feature vector to obtain a target key emotion feature vector. The purpose of feature selection is to determine a feature vector with a high correlation with emotion classification from each target emotion sampling feature vector, that is, to obtain a target key emotion feature vector. Specifically, each target emotion sampling feature vector can be input into a feature selector for feature selection. The feature selector is constructed through a self-attention mechanism. The self-attention mechanism is used to extract key emotion features, thereby improving the accuracy and robustness of speech emotion recognition. In addition, the attention mechanism can be used to explain the target emotion feature extractor's attention and discrimination of different parts of the speech signal, thereby improving the interpretability and visualization of the target emotion feature extractor.

[0130] In one embodiment, referring to Figure 6 , step 301 may include:

[0131] Step 601, performing correlation calculation on each target emotion sampling feature vector and other target emotion sampling feature vectors to obtain target emotion feature correlation;

[0132] Step 602, determining the weight of each target emotion sampling feature vector according to the target emotion feature correlation, and obtaining the target weight;

[0133] Step 603: extracting a target key emotion feature vector from each target emotion sampling feature vector according to the target weight.

[0134] In step 601, a self-attention layer is used to calculate the correlation between each target emotion sampling feature vector and other target emotion sampling feature vectors to obtain the target emotion feature correlation corresponding to each target emotion sampling feature vector.

[0135] In step 602, an activation function may be used to perform weight mapping on the correlation of target emotion features to obtain target weights corresponding to each target emotion sampling feature vector. For example, the activation function is a softmax function.

[0136] In step 603, the higher the target weight, the higher the correlation between the target emotion sampling feature vector and the emotion classification. Specifically, the target emotion sampling feature vectors can be arranged in descending order according to the target weight to obtain the target arrangement order; the target emotion sampling feature vector whose target arrangement order is less than the preset order threshold is determined as the target key emotion feature vector. The target emotion sampling feature vector corresponding to the largest target weight can also be selected as the target key emotion feature vector.

[0137] The benefit of the above-mentioned embodiment of step 601 to step 603 is that the target key emotion feature vector can be determined with high universality.

[0138] In one embodiment, referring to Figure 7 Before step 106, the speech emotion recognition method provided by this embodiment may further include: training to obtain a target classifier, specifically including:

[0139] Step 701, performing loss calculation according to the sample emotion category and the preset second emotion category label to obtain second category loss data;

[0140] Step 702: Adjust the parameters of the initial classifier according to the second category loss data to obtain a target classifier.

[0141] In step 701, the second emotion category label is a multi-label. The second emotion category label indicates the emotion category to which the speech sample signal belongs and the emotion degree of the emotion category. For example, the second emotion category label may include at least one of the following: anger 1, anger 2, anger 3, happy 1, happy 2, happy 3, sad 1, sad 2, sad 3, neutral 1, neutral 2, neutral 3, disgust 1, disgust 2, disgust 3, fear 1, fear 2, fear 3, surprise 1, surprise 2 and surprise 3. Among them, 1, 2, and 3 represent low, medium, and high emotion degrees, respectively. Loss calculation can be performed based on the emotion category and the second emotion category label to obtain second category loss data. The second category loss data can indicate the difference between the sample emotion category and the second emotion category label. The cross entropy loss function can be used when performing loss calculation.

[0142] In step 702, during the parameter adjustment process, the second category loss data is compared with the preset second loss threshold. If the second category loss data is greater than the second loss threshold, the initial classifier continues to be adjusted until the second category loss data is less than or equal to the second loss threshold, then the parameter adjustment is stopped to obtain the target classifier. The Adam optimizer can be used for parameter adjustment, setting the learning rate to 0.001, the batch size to 32, and the number of training rounds to 50. The target classifier can be used to classify the target key emotion feature vector that appears below to obtain the target emotion category.

[0143] The benefit of the embodiment of the above steps 701 to 702 is that the parameters of the initial classifier can be adjusted based on the difference between the sample emotion category and the second emotion category label to obtain the target classifier, which can improve the emotion classification granularity of the target classifier.

[0144] Step 106, classify the target key emotion feature vector by the pre-trained target classifier to obtain the target emotion category. The target classifier can be composed of a fully connected layer. If the initial classifier is directly used as the target classifier, the target classifier has the same structure and parameters as the initial classifier, and the target emotion category can include at least one of the following: anger, happiness, sadness, neutrality, disgust, fear, and surprise. If the target classifier is obtained by training the initial classifier with the second category loss data, the target classifier has the same structure but different parameters as the initial classifier, and the target emotion category label can include at least one of the following: anger 1, anger 2, anger 3, happiness 1, happiness 2, happiness 3, sadness 1, sadness 2, sadness 3, neutral 1, neutral 2, neutral 3, disgust 1, disgust 2, disgust 3, fear 1, fear 2, fear 3, surprise 1, surprise 2, and surprise 3.

[0145] See also Figure 8 The embodiment of the present application also provides a speech emotion recognition device, which can implement the above-mentioned speech emotion recognition method. Figure 8 The module structure block diagram of the speech emotion recognition device provided in the embodiment of the present application includes: a speech acquisition module 801, a feature extraction module 802, a feature transformation module 803, a feature sampling module 804, a feature selection module 805 and an emotion classification module 806. Among them, the speech acquisition module 801 is used to acquire the target speech signal; the feature extraction module 802 is used to extract the spectral feature of the target speech signal to obtain the target spectral feature vector; the feature transformation module 803 is used to perform emotional feature transformation on the target spectral feature vector through a pre-trained target emotion feature extractor to obtain a target emotion potential feature vector; the feature sampling module 804 is used to perform feature sampling on the target emotion potential feature vector to obtain at least two target emotion sampling feature vectors; the feature selection module 805 is used to perform feature selection on each target emotion sampling feature vector to obtain a target key emotion feature vector; the emotion classification module 806 is used to perform emotion classification on the target key emotion feature vector through a pre-trained target classifier to obtain a target emotion category.

[0146] In one embodiment, the speech emotion recognition device further includes: a first training module, used for training to obtain a target emotion feature extractor.

[0147] In another embodiment, the speech emotion recognition device further includes: a first training module, which is also used to update the target loss data.

[0148] In one embodiment, the speech emotion recognition device further includes: a second training module for training to obtain a target classifier.

[0149] It should be noted that the specific implementation of the speech emotion recognition device is basically the same as the specific implementation example of the above-mentioned speech emotion recognition method, and will not be repeated here.

[0150] The embodiment of the present application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned voice emotion recognition method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0151] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0152] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0153] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the speech emotion recognition method of the embodiment of this application;

[0154] Input / output interface 903, used to implement information input and output;

[0155] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0156] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0157] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0158] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned speech emotion recognition method.

[0159] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0160] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0161] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0162] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0164] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0165] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0166] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0167] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0168] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0169] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0170] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for speech emotion recognition, characterized in that: The method comprises: Acquire a target speech signal; Extracting the spectral features of the target speech signal to obtain a target spectral feature vector; Performing an emotional feature transformation on the target sound spectrum feature vector by using a pre-trained target emotion feature extractor to obtain a target emotion latent feature vector; Performing feature sampling on the target emotion potential feature vector to obtain at least two target emotion sampling feature vectors; Performing feature selection on each of the target emotion sampling feature vectors to obtain a target key emotion feature vector; The target key emotion feature vector is subjected to emotion classification by a pre-trained target classifier to obtain a target emotion category.

2. The method according to claim 1, characterized in that Before the pre-trained target emotion feature extractor performs feature transformation on the target sound spectrum feature vector to obtain the target emotion potential feature vector, the method further includes: The target emotion feature extractor is obtained by training, specifically comprising: Acquire the spectral features of the speech sample signal to obtain a sample spectral feature vector; Performing feature transformation on the sample sound spectrum feature vector by a preset initial emotion feature extractor to obtain a sample emotion potential feature vector; Sampling the sample emotion potential feature vector to obtain at least two sample emotion sampling feature vectors; Decoding each of the emotion sampling feature vectors to obtain a speech reconstruction feature vector; wherein the dimension of the speech reconstruction feature vector is the same as the dimension of the sample spectrum feature vector; Perform loss reconstruction according to the speech sample signal and the speech reconstruction feature vector to obtain target loss data; The parameters of the initial emotion feature extractor are adjusted according to the target loss data to obtain the target emotion feature extractor.

3. The method according to claim 2, characterized in that Before adjusting the parameters of the initial emotion feature extractor according to the target loss data to obtain the target emotion feature extractor, the method further includes: Updating the target loss data, specifically including: Performing feature selection on each of the sample emotion sampling feature vectors to obtain a sample key emotion feature vector; Performing emotion classification on the sample key emotion feature vector by using a preset initial classifier to obtain a sample emotion category; Performing loss calculation according to the sample emotion category and a preset first emotion category label to obtain first category loss data; wherein the first emotion category label is a single label, and the first emotion category label indicates the emotion category to which the speech sample signal belongs; The target loss data is updated according to the first category loss data.

4. The method according to claim 3, characterized in that Before performing emotion classification on the target key emotion feature vector by the pre-trained target classifier to obtain the target emotion category, the method further includes: The target classifier is obtained by training, specifically including: Performing loss calculation according to the sample emotion category and a preset second emotion category label to obtain second category loss data; wherein the second emotion category label is a multi-label, and the second emotion category label indicates the emotion category to which the speech sample signal belongs and the emotion degree of the emotion category; The parameters of the initial classifier are adjusted according to the second category loss data to obtain the target classifier.

5. The method according to any one of claims 1 to 4, characterized in that: The target emotion feature extractor includes a target convolution layer and a gated recurrent unit network layer; The target emotion feature extractor is used to perform emotion feature transformation on the target sound spectrum feature vector to obtain the target emotion potential feature vector, including: Performing convolution processing on the target sound spectrum feature vector through the target convolution layer to obtain an initial emotion feature vector; The context feature extraction is performed on the initial emotion feature vector through the gated recurrent unit network layer to obtain the target emotion potential feature vector.

6. The method according to claim 5, characterized in that The target convolution layer includes a first convolution layer, a second convolution layer and a flattening layer, and the convolution kernel size of the first convolution layer is smaller than the convolution kernel size of the second convolution layer; The step of performing convolution processing on the target sound spectrum feature vector through the target convolution layer to obtain an initial emotion feature vector includes: Performing a first convolution on the target sound spectrum feature vector through the first convolution layer to obtain a first emotion initial feature vector; Performing a second convolution on the first initial emotion feature vector through the second convolution layer to obtain a second initial emotion feature vector; The second initial emotion feature vector is transformed into a one-dimensional vector through the flattening layer to obtain the initial emotion feature vector.

7. The method according to any one of claims 1 to 4, characterized in that: The step of performing feature selection on each of the target emotion sampling feature vectors to obtain a target key emotion feature vector includes: Performing correlation calculation on each of the target emotion sampling feature vectors and other target emotion sampling feature vectors to obtain target emotion feature correlation; Determine the weight of each of the target emotion sampling feature vectors according to the target emotion feature correlation to obtain a target weight; The target key emotion feature vector is extracted from each of the target emotion sampling feature vectors according to the target weight.

8. A speech emotion recognition device, characterized in that: The device comprises: A speech acquisition module, used to acquire a target speech signal; A feature extraction module is used to extract the spectral features of the target speech signal to obtain a target spectral feature vector; A feature transformation module, used for performing an emotion feature transformation on the target sound spectrum feature vector by using a pre-trained target emotion feature extractor to obtain a target emotion potential feature vector; A feature sampling module, used for performing feature sampling on the target emotion potential feature vector to obtain at least two target emotion sampling feature vectors; A feature selection module, used to perform feature selection on each of the target emotion sampling feature vectors to obtain a target key emotion feature vector; The emotion classification module is used to perform emotion classification on the target key emotion feature vector through a pre-trained target classifier to obtain a target emotion category.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images