A speech emotion recognition method based on two-stage bottleneck layer feature selection and multi-classifier joint decision algorithm

Abstract

The present invention discloses a speech emotion recognition method based on two-stage bottleneck layer feature selection and multi-classifier joint decision algorithm, including the following steps: S1, extracting Fourier coefficient features and inputting them into a deep neural network, training the deep neural network to extract bottleneck features, and obtaining deep bottleneck layer features and shallow bottleneck layer features; S2, optimizing and fusing deep bottleneck layer features with shallow bottleneck layer features; S3, performing feature screening on the fused deep and shallow bottleneck layer features, reducing feature dimensions, and extracting features with high emotion discrimination; S4, constructing a combined classifier, proposing a multi-classifier joint decision algorithm, performing joint decision, and obtaining the final decision result. The present invention extracts deep bottleneck layer features and shallow bottleneck layer features, and uses GA to fuse these two features, making up for the defect that a single bottleneck layer feature cannot fully express emotional information, and further improving the speech emotion recognition effect.

Description

Technical Field

[0001] The present invention relates to a speech emotion recognition method, in particular to a speech emotion recognition method based on two-stage bottleneck layer feature selection and multi-classifier joint decision (MCJD) algorithm, and relates to the technical field of artificial intelligence. Background Art

[0002] With the popularization of intelligent hardware devices and the advancement of artificial intelligence technology, how to meet the increasingly diverse needs of users has become a key research focus for industry insiders. Language is the most direct and natural way for people to express their wishes. Expanding its application to intelligent hardware devices such as computers so that machines can understand users' needs through language is undoubtedly an extremely convenient technical implementation solution. Therefore, speech recognition technology has been booming. In the process of using language, people will add their own emotions to it, and the investment of different emotions will cause subtle changes in the meaning of the language itself. In order to enable machines to understand users' wishes more accurately and also to detect users' own emotions, speech emotion recognition technology has begun to appear and receive widespread attention.

[0003] At present, the research hotspots of speech emotion recognition technology are mainly concentrated in the following aspects: 1. Speech emotion features, such as related research on non-personalized features and spectrogram features. 2. Classification methods, such as SVM, artificial neural network (ANN), multimodal fusion recognition, etc. 3. Applications of speech emotion recognition, such as its application in different fields of human-computer interaction. In order to further improve the accuracy of speech emotion recognition, it is necessary to comprehensively consider the extraction, selection and fusion of emotional features in speech information and classification methods.

[0004] In current applications, traditional speech emotion features are widely used in speech emotion recognition tasks, in which the acoustic features used mainly include Mel-Frequency Cepstral Coefficients (MFCC), fundamental frequency, zero-crossing rate, and energy amplitude. Although in some cases, the above acoustic features can achieve good performance results, the acoustic features generally only consider the physical level information of the speech signal, and the deeper and richer emotional information has not been fully extracted. Therefore, industry insiders have also tried to replace the acoustic features with bottleneck layer features that can better reflect the deep expression of language emotions. With the continuous deepening of research, Deep Neural Networks (DNN) have shown excellent performance in the extraction of discriminant features. Compared with manual extraction, DNN can extract relevant hierarchical features from a large number of training samples through supervised learning. For example, Schmidt et al. used a deep belief network (DBN) to extract deep emotional features from the amplitude spectrum of speech signals, achieving better recognition performance than traditional acoustic features; Han et al. proposed to use the highest energy speech frames to train the DNN model to extract effective speech emotional features; Mao et al. used convolutional neural networks (CNN) to extract speech emotional features and conducted experiments on several standard data sets, achieving good performance; Lee et al. used recursive neural networks (RNN) to extract the temporal related features of emotions in speech; Trigeorgis et al. directly used the original speech signal to train the convolutional recurrent neural network (CRNN) to predict the continuous arousal-valence space. However, no matter which of the above technical solutions is implemented, there are very obvious defects. One of the main reasons is that it is difficult for a single bottleneck layer feature to fully represent the rich emotional information in speech.

[0005] In addition, classification is another important part of speech emotion recognition, which classifies the speech samples according to the distinguishing features of the extracted speech samples. The most important thing is the selection of classifiers. In short, there are many classification technologies, and the most widely used ones are decision tree technology, KNN, naive Bayes classifier, ANN and SVM. However, a single classification technology will inevitably have certain limitations when performing classification tasks. Therefore, building a good combination classifier has also become a new research branch in speech emotion recognition technology. At present, multi-classifier combination technology has been widely used in many fields such as pattern recognition, network security and word sense disambiguation.

[0006] In summary, if we can propose a new method for speech emotion recognition, focus on the extraction, selection and fusion of emotional features and the improvement of classification methods, use the bottleneck layer features extracted from DNN as speech emotion features, and achieve a high recognition rate in multiple language scenarios, it will definitely provide great help to the future development and application of artificial intelligence technology. Summary of the invention

[0007] In view of the above-mentioned defects in the prior art, the purpose of the present invention is to propose a speech emotion recognition method based on a two-stage bottleneck layer feature selection and a multi-classifier joint decision algorithm, which is specifically as follows.

[0008] A speech emotion recognition method based on two-stage bottleneck layer feature selection and multi-classifier joint decision algorithm includes the following steps:

[0009] S1. Obtain the Fourier coefficient feature of the speech signal and use it as the input of the deep neural network, train multiple different deep neural networks to extract the bottleneck features in the speech signal, thereby obtaining deep bottleneck layer features and shallow bottleneck layer features;

[0010] S2, using genetic algorithm to achieve the optimal fusion of deep bottleneck layer features and shallow bottleneck layer features;

[0011] S3, using the optimal feature subset selection technology to screen the fused deep bottleneck layer features and shallow bottleneck layer features, reduce the feature dimension, and extract features with high emotional discrimination;

[0012] S4. For different classification targets, multiple classifiers are trained and a combined classifier is constructed. A multi-classifier joint decision algorithm is proposed. By jointly judging the results of the combined classifiers, the final decision result of speech emotion is obtained.

[0013] Preferably, S1 comprises the following steps:

[0014] S11, acquiring a speech signal, sampling each sentence of the input time-domain continuous speech signal, and then preprocessing the speech signal using pre-emphasis, frame windowing, and endpoint detection techniques to obtain a preprocessed speech signal;

[0015] S12, calculating the Fourier coefficient features of the preprocessed speech signal and using the obtained Fourier coefficient features as input of the deep neural network;

[0016] S13, setting the positions of bottleneck layers in hidden layers of multiple deep neural networks, and naming them DNN1, DNN2 and DNN3 according to the different positions of bottleneck layers in hidden layers of deep neural networks, wherein DNN1 refers to a deep neural network whose first hidden layer is a bottleneck layer, and so on, performing unsupervised pre-training on DNN1, DNN2 and DNN3 respectively, and then introducing supervised error back propagation to achieve parameter fine-tuning, to obtain trained DNN1, DNN2 and DNN3;

[0017] S14. Re-input the preprocessed speech signal into the trained DNN1, DNN2 and DNN3, extract the features of the corresponding bottleneck layer on each speech frame, and calculate the maximum, minimum, mean, variance and median of the extracted corresponding bottleneck layer features on each speech to obtain the deep and shallow bottleneck layer features of each speech.

[0018] Preferably, S2 comprises the following steps:

[0019] S21, using a genetic algorithm to optimize and fuse the deep bottleneck layer features and the shallow bottleneck layer features, and assigning initial weights to the deep bottleneck layer features and the shallow bottleneck features respectively;

[0020] S22, taking the weighted fusion of the bottleneck layer features by the initial weights as the initialization input of the genetic algorithm, setting the objective function of the genetic algorithm to the average recognition rate and the recognition rate of various target emotions, and starting the genetic algorithm to optimize the fusion weights;

[0021] S23. The genetic algorithm outputs and saves the weight optimization result, and uses the weight optimization result for weighted fusion of deep bottleneck layer features and shallow bottleneck features.

[0022] Preferably, the step of using a genetic algorithm to optimally fuse the deep bottleneck layer features and the shallow bottleneck layer features in S21 comprises the following steps:

[0023] S211, initialize weights, perform binary encoding on weight combinations, and generate an initialized population;

[0024] S212, decoding to obtain a weighted combination, combining features in a weighted manner, importing the combined features into a support vector machine for training, and using the speech emotion recognition result obtained by the support vector machine as a fitness function;

[0025] S213, performing a selection operation, simulating the law of survival of the fittest according to the fitness function, selecting excellent individuals from the population as the parent generation, and generating a new population as the offspring;

[0026] S214, perform mutation operation, randomly select a pair of individuals from a population, and exchange some of their genes to form new individuals;

[0027] S215. For each individual in the population, the gene of the individual is changed with a fixed mutation probability to form a new individual and rejoin the population;

[0028] S216, decoding weights and calculating fitness values, and then comparing speech emotion recognition rates of offspring and parents to update the best individual;

[0029] S217, check whether the number of iterations or the fitness value meets the preset termination condition. If not, repeat steps S213 to S216. If satisfied, proceed to step S218 in sequence.

[0030] S218. Output the optimal weight combination.

[0031] Preferably, S3 includes the following steps:

[0032] S31. Based on the fused deep bottleneck layer features and shallow bottleneck layer features, the optimal feature subset selection technology is used to perform feature screening, and different dimensionality reduction features and corresponding emotion recognition rates are obtained by setting different thresholds;

[0033] S32. According to the different dimensionality reduction features and corresponding emotion recognition rates obtained in S31, a feature set with the highest emotion recognition rate is selected as a result of feature screening.

[0034] Preferably, the step of using the optimal feature subset selection technology to perform feature screening based on the fused deep bottleneck layer features and shallow bottleneck layer features in S31 includes the following steps:

[0035] S311, given input sample data X = (x1, x2, ..., x M ), calculate its covariance matrix

[0036] S312, finding all eigenvalues ​​and corresponding eigenvectors of the covariance matrix, and performing standardization processing on all eigenvectors;

[0037] S313, sort the eigenvalues ​​of the solved covariance matrix from large to small, k1>k2>…>k M >0;

[0038] S314. If the data dimension is to be reduced to K dimensions, the first K largest eigenvalues ​​are taken, and the eigenvectors obtained in S312 are arranged into a matrix in the order of the corresponding eigenvalues, forming a matrix W = (w1, w2, ..., w K );

[0039] S315, mapping to a new sample, Z = WX, which is the data after dimension reduction to K dimension.

[0040] Preferably, S4 comprises the following steps:

[0041] S41, using the fused deep bottleneck layer features and shallow bottleneck layer features to train the support vector machine classifier to obtain eight basic classifiers SVM0 to SVM7, where the classification target of SVM0 is the average emotion recognition rate, and the classification target of SVM1 to SVM7 is the recognition rate of the i-th target emotion, where i is an integer between 1 and 7;

[0042] S42, forming a new combined classifier, making a final decision on the category of speech emotion based on the multi-classifier joint decision algorithm. When the output of SVM0 is i emotions, the process based on the multi-classifier joint decision algorithm is:

[0043] If the predicted emotion of SVM0 is consistent with that of SVMi, this emotion is selected as the final judgment result of the classifier;

[0044] If the predicted emotion of SVM0 is inconsistent with the predicted emotion of SVMi, at this time, if there is a basic classifier in SVM1~SVM7 whose prediction result is its corresponding target emotion, then among these basic classifier results, the emotion corresponding to the basic classifier with the highest probability of target emotion is selected as the final judgment result of the classifier. If there is none, the output of SVM0 is selected as the judgment result of the classifier.

[0045] The advantages of the present invention are mainly reflected in:

[0046] The speech emotion recognition method proposed in the present invention, which is based on two-stage bottleneck layer feature selection and multi-classifier joint decision algorithm, overcomes the shortcomings of other speech emotion recognition methods in the prior art in many aspects. Compared with a single bottleneck layer feature, the present invention extracts deep bottleneck layer features and shallow bottleneck layer features, and uses GA to fuse these two features, thereby making up for the defect that a single bottleneck layer feature cannot fully express emotional information, and further improving the speech emotion recognition effect. At the same time, compared with the traditional SVM classifier, the present invention constructs multiple basic classifiers according to different classification objectives, and realizes speech emotion classification through the MCJD algorithm, which greatly improves the system performance and can achieve higher emotion recognition performance in different language scenarios.

[0047] In addition, the solution of the present invention also provides a new idea for the related research and application of speech emotion recognition technology, and provides a reference for other related issues in the same field. It can be used as a basis for expansion, extension and in-depth research, and has a very broad application prospect.

[0048] The specific implementation modes of the present invention are further described below in combination with the accompanying drawings of the embodiments, so that the technical solutions of the present invention are easier to understand and grasp. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The flowchart of the speech emotion recognition system of the method of the present invention is as follows;

[0050] Figure 2 is a schematic diagram of a DNN model used to extract bottleneck features in the present invention;

[0051] Figure 3 It is a two-stage feature selection flow chart based on GA in the present invention;

[0052] Figure 4 is a flow chart of the MCJD algorithm in the present invention;

[0053] Figure 5 Schematic diagram of average recognition rate of speech emotion based on two-stage bottleneck layer feature selection in the present invention;

[0054] Figure 6 It is a schematic diagram of the comparison results (DNN1+DNN2) before and after PCA dimensionality reduction in the present invention;

[0055] Figure 7 It is a schematic diagram of the performance comparison of the bottleneck layer feature combined with the MCJD algorithm and the combined basic SVM classifier in the present invention. DETAILED DESCRIPTION

[0056] The present invention discloses a speech emotion recognition method based on two-stage bottleneck layer feature selection and multi-classifier joint decision algorithm. Figure 1 The flowchart of the speech emotion recognition system of the present invention is as follows. In practical applications, the acoustic features of the speech signal are generally extracted, the acoustic features are input into the classifier for training, and then the trained classifier model is used for emotion recognition. In recent years, deep learning has achieved many results in data mining, pattern recognition, natural language processing, multimedia learning, speech recommendation and personalization technology, and other related fields. Its powerful feature extraction and modeling capabilities have greatly improved the performance of pattern recognition. Therefore, the present invention applies deep neural networks to speech emotion recognition to extract deep and shallow bottleneck layer features, and can build a higher performance speech emotion recognition system.

[0057] In order to effectively improve the performance of speech emotion recognition, the present invention proposes a speech emotion recognition method based on two-stage bottleneck layer feature selection. In the feature selection of the first stage, appropriate bottleneck layer features are extracted from a deep neural network, and feature fusion based on a genetic algorithm is achieved. A large number of experiments in the field of speech recognition have proven that the bottleneck layer features extracted using a deep neural network have good robustness and discriminability. The genetic algorithm can find the global optimal solution of an optimization problem and has strong robustness. Therefore, the bottleneck layer fusion features based on the genetic algorithm have good robustness. In the feature selection of the second stage, an optimal feature subset selection technique is adopted to eliminate the curse of dimensionality problem. Basically, feature selection is the process of finding an n < N-dimensional feature subset from a given N-dimensional feature set without significantly degrading the performance of the classifier. In feature subset selection, each feature is assigned a value reflecting its usefulness.

[0058] However, the most suitable feature combination for each emotion is different. Therefore, for each emotion, the uniqueness of each emotion category should be taken into account. Different fitness functions are selected for each emotion, and the genetic algorithm separately performs parameter optimization of the contribution weights. After obtaining different weight sets, the deep and shallow bottleneck layer features are fused. In this way, the obtained fused feature sets for each group will be more conducive to improving the recognition rate of the corresponding emotion category. Based on this idea, the present invention proposes an MCJD algorithm to make a final decision on speech emotions, further enhancing the robustness of the speech emotion recognition system. The following is a detailed discussion of the specific implementation manner of the present invention.

[0059] A speech emotion recognition method based on two-stage bottleneck layer feature selection and multi-classifier joint decision algorithm of the present invention includes the following steps:

[0060] S1. Obtain the Fourier coefficient features of the speech signal and use them as the input of a deep neural network (DNN). Train multiple different deep neural networks to extract the bottleneck features in the speech signal, so as to obtain deep bottleneck layer features and shallow bottleneck layer features.

[0061] S1 specifically includes the following steps:

[0062] S11. Obtain the speech signal, sample each input time-domain continuous speech signal, and then pre-emphasize, frame and window, and perform endpoint detection techniques on the speech signal respectively to obtain the preprocessed speech signal;

[0063] S12. Calculate the Fourier coefficient features of the preprocessed speech signal and use the obtained Fourier coefficient features as the input of a deep neural network (DNN);

[0064] S13, setting the position of the bottleneck layer in the hidden layers of multiple deep neural networks. The DNN used in the present invention is a structure including five layers of networks, such as Figure 2 As shown. The input layer is a 1280-dimensional Fourier feature vector after normalization. There are three hidden layers, one of which has significantly fewer neurons than the other hidden layers, and is the bottleneck layer. The number of neurons in this layer is set to 100, and the number of neurons in the other hidden layers is set to 1280. Softmax is used as the output layer of DNN, and the number of neurons in this layer is consistent with the classification category. According to the different positions of the bottleneck layer in the hidden layer, the present invention uses three DNNs, which are named DNN1, DNN2 and DNN3, respectively, where DNN1 refers to a deep neural network with the first hidden layer as the bottleneck layer, and so on. The input layer and the output layer remain unchanged, and are used to extract bottleneck layer features based on different hidden layers. Among them, on the EMO-DB corpus, the network structure of DNN1 is 1280-100-1280-1280-7, the network structure of DNN2 is 1280-1280-100-1280-7, and the network structure of DNN3 is 1280-1280-1280-100-7. Unsupervised pre-training is performed on DNN1, DNN2, and DNN3, and then supervised error back propagation is introduced to achieve parameter fine-tuning, and the trained DNN1, DNN2, and DNN3 are obtained;

[0065] S14. Re-input the preprocessed speech signal into the trained DNN1, DNN2 and DNN3, extract the features of the corresponding bottleneck layer on each speech frame, and calculate the maximum value, minimum value, mean value, variance and median of the extracted corresponding bottleneck layer features on each speech, a total of five global statistics, to obtain the deep and shallow bottleneck layer features of each speech.

[0066] S2. Genetic Algorithm (GA) is used to optimize the fusion of deep bottleneck layer features and shallow bottleneck layer features to make up for the defect that a single bottleneck layer feature cannot fully express emotional information, and further improve the effect of speech emotion recognition.

[0067] S2 specifically includes the following steps:

[0068] S21, using a genetic algorithm to optimize and fuse the deep bottleneck layer features and the shallow bottleneck layer features, and assigning initial weights to the deep bottleneck layer features and the shallow bottleneck features respectively;

[0069] S22, taking the weighted fusion of the bottleneck layer features by the initial weights as the initialization input of the genetic algorithm, setting the objective function of the genetic algorithm to the average recognition rate and the recognition rate of various target emotions, and starting the genetic algorithm to optimize the fusion weights;

[0070] S23. The genetic algorithm outputs and saves the weight optimization result, and uses the weight optimization result for weighted fusion of deep bottleneck layer features and shallow bottleneck features.

[0071] Furthermore, the use of a genetic algorithm in S21 to optimize and fuse the deep bottleneck layer features with the shallow bottleneck features includes the following steps:

[0072] S211, initialize weights, perform binary encoding on weight combinations, and generate an initialized population;

[0073] S212, decoding to obtain a weighted combination, combining features in a weighted manner, importing the combined features into a support vector machine for training, and using the speech emotion recognition result obtained by the support vector machine as a fitness function, wherein the higher the fitness value, the greater the possibility of retaining the individual;

[0074] S213, performing a selection operation, simulating the law of survival of the fittest according to the fitness function, selecting excellent individuals (a set of weights represents one individual) from the population as the parent generation, and generating a new population as the offspring;

[0075] S214, perform mutation operation, randomly select a pair of individuals from a population, and exchange some of their genes to form new individuals;

[0076] S215. For each individual in the population, the gene of the individual is changed with a fixed mutation probability to form a new individual and rejoin the population;

[0077] S216, decoding weights and calculating fitness values, and then comparing speech emotion recognition rates of offspring and parents to update the best individual;

[0078] S217, check whether the number of iterations or the fitness value meets the preset termination condition. If not, repeat steps S213 to S216. If satisfied, proceed to step S218 in sequence.

[0079] S218. Output the optimal weight combination.

[0080] The above step S2 can be understood to include the following operations using expressions related to numerical values ​​and functions.

[0081] Assume that the i-th feature in the feature set is represented by x i , the contribution weight of the i-th feature is w i The present invention fuses the deep bottleneck layer features and the shallow bottleneck layer features. The fused feature set can be expressed as X = [w1*x1, w2*x2].

[0082] The present invention uses GA to weightedly fuse the characteristics of the bottleneck layer based on the deep and shallow layers of different hidden layers. When GA is used to search for the optimal w1 and w2. X needs to be used as the input of the genetic algorithm, and then randomly initialized to generate multiple groups of {w1, w2} individuals. These individuals constitute the initialized population P(0), and then these {w1, w2} are encoded according to the GA rules. The present invention adopts a binary encoding method, selects a binary string to represent each parameter, and establishes a relationship with the parameters. Then the binary strings are connected to form a long binary string, which is an object that can be operated by the genetic algorithm. Assume that the binary string sequence is a binary encoding form of {w1, w2}, these binary string sequences are called chromosomes in the genetic algorithm, and {w1, w2} represents an individual. Then the encoded chromosome is decoded to obtain the weight parameters that need to be optimized in the individual, and the optimized parameters are placed in the fitness function to calculate the fitness value of the individual. Then it is determined whether the number of iterations is reached. If it is reached, the search is terminated and the optimized weight value is output. Otherwise, the selection and crossover operations of the genetic algorithm are continued.

[0083] The process of optimizing parameters through genetic algorithms is actually the process of solving the extreme value of the function, and finally obtaining the optimal feature weight combination. The two-stage feature selection process based on genetic algorithms is as follows: Figure 3 The specific steps are as follows:

[0084] A. First, preprocess the speech signal, train the DNN model, and extract the deep and shallow bottleneck layer features;

[0085] B. Determine the parameter encoding scheme and encode the feature weight related parameters in binary encoding mode;

[0086] C. Determine the fitness function;

[0087] D. Genetic algorithm optimization operation;

[0088] E. After optimization by genetic algorithm, the optimal parameters of feature weights are obtained, so that the deep and shallow bottleneck layer features are weighted and fused;

[0089] F. After feature dimensionality reduction, the classifier predicts the final emotion category.

[0090] S3. The optimal feature subset selection technology (PCA) is used to screen the fused deep bottleneck layer features and shallow bottleneck layer features. The use of the optimal feature subset selection technology here not only reduces the amount of input information and reduces feature redundancy, obtains high-quality features with low redundancy and high sentiment discrimination, but also converts the input set into a set of ordered independent components.

[0091] S3 specifically includes the following steps:

[0092] S31. Based on the fused deep bottleneck layer features and shallow bottleneck layer features, the optimal feature subset selection technology is used to perform feature screening, and different dimensionality reduction features and corresponding emotion recognition rates are obtained by setting different thresholds;

[0093] S32. According to the different dimensionality reduction features and corresponding emotion recognition rates obtained in S31, a feature set with the highest emotion recognition rate is selected as a result of feature screening.

[0094] Furthermore, the method described in S31 uses the optimal feature subset selection technology to perform feature screening based on the fused deep bottleneck layer features and shallow bottleneck layer features, including the following steps:

[0095] S311, given input sample data X = (x1, x2, ..., x M ), calculate its covariance matrix

[0096] S312, finding all eigenvalues ​​and corresponding eigenvectors of the covariance matrix, and performing standardization processing (basis vectors) on all eigenvectors;

[0097] S313, sort the eigenvalues ​​of the solved covariance matrix from large to small, k1>k2>...>k M >0;

[0098] S314. If the data dimension is to be reduced to K dimensions, the first K largest eigenvalues ​​are taken, and the eigenvectors obtained in S312 are arranged into a matrix in the order of the corresponding eigenvalues, forming a matrix W = (w1, w2, ..., w K );

[0099] S315, mapping to a new sample, Z = WX, which is the data after dimension reduction to K dimension.

[0100] S4. For different classification targets, multiple classifiers are trained and combined classifiers are constructed. A multi-classifier joint decision (MCJD) algorithm is proposed to obtain the final decision result of the combined classifier. Taking the construction of the basic classifier on the EMO-DB corpus as an example, if the feature weight set is obtained by the genetic algorithm with the average emotion recognition rate as the fitness function, the weighted fusion bottleneck layer feature set is the global expression of the seven emotions. The SVM classification model trained in this way focuses on the overall performance of the seven emotions, which we call SVM0. The other seven basic classifiers focus on the performance of a certain target emotion, ignore other emotions, and focus on one target emotion. For SVM1~SVM7, the recognition rate of a target emotion is used as the fitness function of the genetic algorithm to obtain the contribution weight of the emotion feature. Then the weighted fusion features are input into the traditional support vector machine classifier, and the SVM1~SVM7 classification models corresponding to anger, happiness, boredom, calmness, sadness, fear and disgust are trained respectively. The training objectives of each basic classifier are shown in Table 1.

[0101] Table 1. Training objectives of basic classifiers

[0102]

[0103]

[0104] It can be seen from Table 1 that the training objectives of the basic classifier and the selection of the fitness function of the corresponding GA are consistent. In this way, the feature set obtained directly from the classification results is more conducive to improving the recognition performance of the classifier.

[0105] S4 specifically includes the following steps:

[0106] S41, using the fused deep bottleneck layer features and shallow bottleneck layer features to train a support vector machine (SVM) classifier to obtain eight basic classifiers SVM0 to SVM7, where the classification target of SVM0 is the average emotion recognition rate, and the classification target of SVM1 to SVM7 is the recognition rate of the i-th target emotion, where i is an integer between 1 and 7;

[0107] S42. Since a single classifier does not necessarily have a good recognition effect for each emotion, a new combined classifier is further formed based on the basic classifier proposed in step S41, and the final judgment on the category of speech emotion is made based on the multi-classifier joint judgment algorithm. The purpose of the proposed MCJD algorithm is to further improve the speech emotion recognition rate based on SVM0. The algorithm flow chart is as follows: Figure 4As shown, first check the output of SVM0. If the output of SVM0 is the angry emotion category, then find SVM1 (SVM1 is a classifier that uses the angry emotion recognition rate as the training target, so among SVM0~SVM7, SVM1 has the best recognition effect on angry speech). If the output of SVM1 is also the angry emotion category, then the classification result is angry emotion. If not, you need to check the outputs of SVM1~SVM7 one by one. If there is a classifier that is judged on its target category, then choose the one with the highest probability as the classification result; if not, choose the output of SVM0 as the classification result. If the output of SVM0 is the other six emotions, similar decisions can be made, which will not be repeated here. That is, when the output of SVM0 is i emotions, the process based on the multi-classifier joint judgment algorithm is,

[0108] If the predicted emotion of SVM0 is consistent with that of SVMi, this emotion is selected as the final judgment result of the classifier.

[0109] If the predicted emotion of SVM0 is inconsistent with the predicted emotion of SVMi, at this time, if there is a basic classifier in SVM1~SVM7 whose prediction result is its corresponding target emotion, then among these basic classifier results, the emotion corresponding to the basic classifier with the highest probability of target emotion is selected as the final judgment result of the classifier. If there is none, the output of SVM0 is selected as the judgment result of the classifier.

[0110] Finally, the performance of the above method flow of the present invention is experimentally demonstrated, and the specific operations are as follows.

[0111] The experiment was conducted on the CASIA speech database and the EMO-DB speech database. In the experiment, the ratio of the number of training samples to test samples was 8:2. Since the results of an experiment are too accidental, the experiments were all conducted using the five-fold cross-validation method to obtain a more accurate recognition rate for each corpus. The experiment was conducted using the LIBSVM tool in the MATLAB R2018a environment. In the experiment, the CPU model of the computer was Intel Core i5-8250U and the graphics card model was NVIDIA GeForce MX150.

[0112] First, in order to select features that are more beneficial to sentiment classification, the PCA feature screening method is used to reduce the dimension of the feature set before training the SVM classifier model. Using the reduced-dimensional features to train the classifier can effectively reduce the training time of the classifier. At the same time, the trained classifier has better recognition performance for speech emotions.

[0113] Figure 5The average recognition rate of the bottleneck layer features of different DNN hidden layers after fusion on the CASIA and EMO-DB corpora is summarized after adopting the two-stage bottleneck layer feature selection model. After PCA dimensionality reduction, the emotion recognition rate of the fused bottleneck layer features is improved. Among them, the bottleneck layer features of the fused DNN1 and DNN2 performed best in terms of emotion recognition rate. On the EMO-DB corpus, the average recognition rate after dimensionality reduction is 84.89%, which is 3.78% higher than the recognition rate before dimensionality reduction. Experiments have shown that the combined features of the bottleneck layer features of DNN1 and DNN2 perform best in the speech emotion recognition task.

[0114] Next, the experiment uses the combination of bottleneck layer features of DNN1 and DNN2 as sentiment features. Figure 6 The classification results of 7 emotions on the EMO-DB corpus are summarized with and without PCA dimensionality reduction fusion features. Figure 6 As shown in the figure, except for the sad emotion, the recognition rates of the other six emotions have been greatly improved. Among them, the recognition rate of the happy emotion has increased by 6.74%. The experimental results verify the effectiveness of the PCA algorithm in bottleneck feature selection.

[0115] Finally, the effectiveness of the MCJD algorithm is verified. In the process of constructing the basic classifier, in order to obtain different feature weight sets, different fitness functions need to be applied in the GA optimization process to further obtain basic classifiers suitable for different classification tasks. The optimal weights of the bottleneck layer features searched by GA are shown in Table 2.

[0116] Table 2 Optimal bottleneck layer feature weights corresponding to SVMs

[0117]

[0118] As can be seen from Table 2, the optimal bottleneck layer feature weight set corresponding to each basic classifier based on SVM is different. Taking SVM0 as an example, the weight set obtained by GA optimization is {0.1280, 0.8404}, so the feature set of SVM0 is {0.1280x1, 0.8404x2}. Different basic classifiers SVM0-SVM7 are obtained by training with these different feature sets, and then the primary prediction results of these basic classifiers are integrated through the MCJD algorithm to obtain the final judgment result.

[0119] After the basic classifier is constructed, the MCJD algorithm is used to make a final judgment on the classification result. Table 3 shows the speech emotion recognition results based on the two-stage bottleneck layer feature selection model and the MCJD algorithm proposed in the present invention on the EMO-DB corpus.

[0120] Table 3 Speech emotion recognition rate (%) based on two-stage bottleneck layer feature selection and MCJD algorithm (EMO-DB corpus)

[0121]

[0122]

[0123] It can be seen that the average recognition rate is 87.64%, which is 2.75% higher than the average recognition rate using the SVM classifier, verifying the effectiveness of the two-stage bottleneck feature selection model and the MCJD algorithm. The improvement in speech emotion recognition rate using the MCJD algorithm compared to the basic SVM classifier is shown in the figure below. Figure 7 As shown in the figure, by integrating the judgment results of 8 basic classifiers through the MCJD algorithm, the final classification results have been improved in 7 categories of emotions. Although the recognition of boredom and sadness is still slightly insufficient, the recognition rate of these two emotions is effectively improved through the MCJD algorithm. This proves to a certain extent that the classification system based on the new MCJD algorithm can obtain better recognition results than the traditional SVM classification.

[0124] The above results show that: the present invention proposes two methods to improve the overall recognition rate of the speech emotion recognition system from the two aspects of feature selection and classification model construction. The experimental results verify the rationality and effectiveness of these two methods. Compared with other methods, the method of the present invention can greatly improve the recognition rate, overcome the shortcomings of ordinary methods in not fully describing speech emotion information, and has a good reference significance in practical applications.

[0125] In summary, compared with the prior art, the advantages of the present invention are mainly reflected in the following aspects:

[0126] 1. The speech emotion recognition method of the present invention is based on two-stage bottleneck layer feature selection, which has certain theoretical research value and practical application value. This method extracts the deep and shallow bottleneck layer features of the speech signal and fuses them to make up for the shortcoming that a single feature cannot fully represent the speech emotion information, so that the system performance is greatly improved and a higher recognition rate can be achieved in different language contexts.

[0127] 2. The present invention selects features with high contribution to emotion recognition by using PCA to screen features, while reducing the redundancy of features and the overall computational complexity of the system.

[0128] 3. The present invention adopts GA to optimize and fuse the features. Compared with the simple series fusion method, the GA-based optimal fusion can achieve a higher recognition rate and improve the speech emotion recognition performance.

[0129] 4. The speech emotion recognition method of the present invention also involves a multi-classifier joint decision algorithm. Since a single classifier does not necessarily have a good recognition effect on every emotion, the present invention, on the basis of constructing basic classifiers, combines their primary prediction results to form a new combined classifier to make a final decision on the category of speech emotion, thereby further improving the speech emotion recognition performance.

[0130] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit and essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

[0131] Finally, it should be understood that although this specification is described according to implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A speech emotion recognition method based on two-stage bottleneck layer feature selection and multi-classifier joint decision algorithm, characterized in that: The steps include: S1. Obtain the Fourier coefficient feature of the speech signal and use it as the input of the deep neural network, train multiple different deep neural networks to extract the bottleneck features in the speech signal, thereby obtaining deep bottleneck layer features and shallow bottleneck layer features; S11, acquiring a speech signal, sampling each sentence of the input time-domain continuous speech signal, and then preprocessing the speech signal using pre-emphasis, frame windowing, and endpoint detection techniques to obtain a preprocessed speech signal; S12, calculating the Fourier coefficient features of the preprocessed speech signal and using the obtained Fourier coefficient features as input of the deep neural network; S13, setting the positions of bottleneck layers in hidden layers of multiple deep neural networks, and naming them DNN1, DNN2 and DNN3 according to the different positions of bottleneck layers in hidden layers of deep neural networks, wherein DNN1 refers to a deep neural network whose first hidden layer is a bottleneck layer, and so on, performing unsupervised pre-training on DNN1, DNN2 and DNN3 respectively, and then introducing supervised error back propagation to achieve parameter fine-tuning, to obtain trained DNN1, DNN2 and DNN3; S14, re-input the pre-processed speech signal into the trained DNN1, DNN2 and DNN3, extract the features of the corresponding bottleneck layer on each speech frame, and calculate the maximum value, minimum value, mean value, variance and median of the extracted corresponding bottleneck layer features on each speech, so as to obtain the deep and shallow bottleneck layer features of each speech; S2, using genetic algorithm to achieve the optimal fusion of deep bottleneck layer features and shallow bottleneck layer features; S3, using the optimal feature subset selection technology to screen the fused deep bottleneck layer features and shallow bottleneck layer features, reduce the feature dimension, and extract features with high emotional discrimination; S4. For different classification targets, multiple classifiers are trained and combined classifiers are constructed. A multi-classifier joint decision algorithm is proposed. By making joint decisions on the results of the combined classifiers, the final decision result of speech emotion is obtained. S41, using the fused deep bottleneck layer features and shallow bottleneck layer features to train the support vector machine classifier to obtain eight basic classifiers SVM0 to SVM7, where the classification target of SVM0 is the average emotion recognition rate, and the classification target of SVM1 to SVM7 is the recognition rate of the i-th target emotion, where i is an integer between 1 and 7; S42, forming a new combined classifier, making a final decision on the category of speech emotion based on the multi-classifier joint decision algorithm. When the output of SVM0 is i emotions, the process based on the multi-classifier joint decision algorithm is: If the predicted emotion of SVM0 is consistent with that of SVMi, this emotion is selected as the final judgment result of the classifier; If the predicted emotion of SVM0 is inconsistent with the predicted emotion of SVMi, at this time, if there is a basic classifier in SVM1~SVM7 whose prediction result is its corresponding target emotion, then among these basic classifier results, the emotion corresponding to the basic classifier with the highest probability of target emotion is selected as the final judgment result of the classifier. If there is none, the output of SVM0 is selected as the judgment result of the classifier.

2. According to claim 1, a speech emotion recognition method based on two-stage bottleneck layer feature selection and multi-classifier joint decision algorithm is characterized in that: S2 includes the following steps: S21, using a genetic algorithm to optimize and fuse the deep bottleneck layer features and the shallow bottleneck layer features, and assigning initial weights to the deep bottleneck layer features and the shallow bottleneck features respectively; S22, taking the weighted fusion of the bottleneck layer features by the initial weights as the initialization input of the genetic algorithm, setting the objective function of the genetic algorithm to the average recognition rate and the recognition rate of various target emotions, and starting the genetic algorithm to optimize the fusion weights; S23. The genetic algorithm outputs and saves the weight optimization result, and uses the weight optimization result for weighted fusion of deep bottleneck layer features and shallow bottleneck features.

3. A speech emotion recognition method based on two-stage bottleneck layer feature selection and multi-classifier joint decision algorithm according to claim 2, characterized in that: The use of a genetic algorithm to optimize and fuse the deep bottleneck layer features with the shallow bottleneck features described in S21 includes the following steps: S211, initialize weights, perform binary encoding on weight combinations, and generate an initialized population; S212, decoding to obtain a weighted combination, combining features in a weighted manner, importing the combined features into a support vector machine for training, and using the speech emotion recognition result obtained by the support vector machine as a fitness function; S213, performing a selection operation, simulating the law of survival of the fittest according to the fitness function, selecting excellent individuals from the population as the parent generation, and generating a new population as the offspring; S214, perform mutation operation, randomly select a pair of individuals from a population, and exchange some of their genes to form new individuals; S215. For each individual in the population, the gene of the individual is changed with a fixed mutation probability to form a new individual and rejoin the population; S216, decoding the weights and calculating the fitness value, and then comparing the speech emotion recognition rates of the offspring and the parent to update the best individual; S217, check whether the number of iterations or the fitness value meets the preset termination condition. If not, repeat steps S213 to S216. If satisfied, proceed to step S218 in sequence. S218. Output the optimal weight combination.

4. The method for speech emotion recognition based on two-stage bottleneck layer feature selection and multi-classifier joint decision algorithm according to claim 2 is characterized in that: S3 includes the following steps: S31. Based on the fused deep bottleneck layer features and shallow bottleneck layer features, the optimal feature subset selection technology is used to perform feature screening, and different dimensionality reduction features and corresponding emotion recognition rates are obtained by setting different thresholds; S32. According to the different dimensionality reduction features and corresponding emotion recognition rates obtained in S31, a feature set with the highest emotion recognition rate is selected as a result of feature screening.

5. The method for speech emotion recognition based on two-stage bottleneck layer feature selection and multi-classifier joint decision algorithm according to claim 4 is characterized in that: According to S31, the optimal feature subset selection technology is used to perform feature screening based on the fused deep bottleneck layer features and shallow bottleneck layer features, including the following steps: S311, given input sample data X=(x1, x2, ..., x M ), calculate its covariance matrix S312, finding all eigenvalues ​​and corresponding eigenvectors of the covariance matrix, and performing standardization processing on all eigenvectors; S313, sort the eigenvalues ​​of the solved covariance matrix from large to small, k1>k2>...>k M >0; S314. If the data dimension is to be reduced to K dimensions, the first K largest eigenvalues ​​are taken, and the eigenvectors obtained in S312 are arranged into a matrix in the order of the corresponding eigenvalues, forming a matrix W = (w1, w2, ..., w K ); S315, mapping to a new sample Z=WX, that is, the data after dimension reduction to K dimension.

Images