A method, system, storage medium, and electronic device for voice emotion recognition.

By combining K-means clustering and the EfficientNet-B1 model, the data imbalance problem in unlabeled data processing in semi-supervised speech emotion recognition is solved, improving the recognition accuracy of the model, especially on the IEMOCAP-4 database, achieving higher speech emotion recognition performance.

CN118098285BActive Publication Date: 2026-11-10CHINA UNIV OF GEOSCIENCES (WUHAN)
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410183246.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-19
Publication Date
2026-11-10
Estimated Expiration
2044-02-19

AI Technical Summary

Technical Problem

Existing semi-supervised speech emotion recognition methods suffer from data imbalance when processing unlabeled data. They ignore the specificity of emotion categories, leading to inconsistent labels from different annotators during the annotation process, which affects the model's generalization ability and recognition accuracy.

Method used

By analyzing the feature distribution of spectrogram data using K-means clustering, and combining it with the EfficientNet-B1 model for feature extraction and pre-training, and using pseudo-labels and loss function constraints to filter and enhance unlabeled data, a semi-supervised speech emotion recognition system based on K-means clustering is constructed to alleviate the data imbalance problem and improve model performance.

Benefits of technology

Effective use of unlabeled data improves the model's generalization ability and recognition accuracy, especially on the IEMOCAP-4 database, where the recognition accuracy increases from 67.98% to 71.11%, and on EMODB from 89.09% to 90.40%.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118098285B_ABST
    Figure CN118098285B_ABST
Patent Text Reader

Abstract

This invention discloses a speech emotion recognition method, system, storage medium, and electronic device, relating to the field of speech emotion recognition technology. The speech emotion recognition method includes: extracting features from labeled spectrogram data and performing K-means clustering; pre-training a classification model using the probability proportion of emotion categories in the obtained clusters and the labeled spectrogram data; using the feature extraction model trained on the labeled spectrogram data to extract features from unlabeled spectrogram data; fitting the K-means clustering using the labeled data; filtering unlabeled data based on the new cluster centers and features of the unlabeled spectrogram data; performing weak enhancement on the labeled data, and weak and strong enhancement on the filtered unlabeled data; inputting all these into the pre-trained classification model for training, and using a loss function for constraint, to obtain the final classification model for speech emotion recognition. This method has high recognition accuracy and good feasibility.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of voice emotion recognition technology, and in particular to a voice emotion recognition method, system, storage medium, and electronic device. Background Technology

[0002] Speech emotion recognition methods are mainly divided into two parts: emotion feature extraction-based speech emotion recognition and spectrogram classification-based speech emotion recognition. Current research extensively combines both methods with algorithms from the field of deep learning to achieve better recognition results. These two methods will be described in detail below.

[0003] ① Speech emotion recognition based on emotion feature extraction

[0004] These speech emotion recognition methods generally include audio signal preprocessing, emotion feature extraction, and emotion classification. Emotion feature extraction is the most crucial step because the acoustic characteristics of speech signals are highly complex, and not all features are related to emotion. Therefore, identifying the feature parameters that reflect emotional differences directly impacts the final recognition accuracy. Common speech emotion features include spectral features, prosodic features, phonological features, and cutting-edge speech emotion features. Generally, the extracted emotion features need dimensionality reduction before traditional machine learning methods, such as SVM, KNN, and HMM, are used for emotion classification. Traditionally, emotion feature extraction was done manually; current methods tend to automate the process using deep learning algorithms to avoid the significant uncertainties inherent in manual emotion feature extraction.

[0005] ② Speech emotion recognition based on spectrogram classification

[0006] Spectrogram-based speech emotion recognition methods typically transform the problem into an image classification problem after preprocessing the audio signal and converting it into a spectrogram. Deep learning algorithms are then used to build a model for emotion classification. These methods generally achieve better accuracy, but they are computationally expensive and rely on large-scale data for model training. However, in practical applications, large-scale, labeled data is not always readily available. Therefore, using semi-supervised learning methods for speech emotion recognition is of great significance.

[0007] In the real world, labeled data is often more difficult to obtain than unlabeled data. Traditional supervised learning algorithms require a large amount of labeled data to train the model, but obtaining labeled data is costly; therefore, semi-supervised learning emerged. Semi-supervised learning aims to improve model performance by utilizing limited labeled data and a large amount of unlabeled data. By leveraging information from unlabeled data, semi-supervised learning can better utilize the latent structure and distribution of the data, thereby improving the model's generalization ability. Existing semi-supervised learning methods can be categorized into four types: entropy minimization-based methods, generative methods, consistent regularization-based methods, and hybrid methods.

[0008] The basic idea behind entropy minimization methods is that the predicted output of unlabeled data should be as "deterministic" as possible. A typical strategy is the pseudo-labeling method, but research relying solely on pseudo-labeling is limited. Entropy minimization can be combined with other learning methods. Some studies combine self-supervised pseudo-label classification networks with supervised regression models to create an end-to-end speech emotion recognition framework, while encouraging effective separation of emotion features. Furthermore, methods that address privacy and data processing efficiency combine federated learning and self-training to fully utilize both labeled and unlabeled data on the device, both of which improve speech emotion recognition performance. The advantage of this approach lies in its ability to effectively utilize the latent information in unlabeled data, enhancing the model's generalization ability.

[0009] Generative methods adhere to the assumption of generative models. They aim to leverage both labeled and unlabeled data by modeling the joint distribution. These methods typically assume that the data is generated by an underlying generative process and attempt to understand the data's structure by modeling this process. Current research in this area applied to speech emotion recognition is primarily based on semi-supervised autoencoder frameworks.

[0010] Consistent regularization is a technique that leverages unlabeled data to enhance the performance of learning models. The core concept is that a good model should produce stable predictions when the input data is slightly perturbed; that is, the model's output should remain consistent despite small changes in the input data. In speech emotion recognition, consistent regularization is primarily applied using trapezoidal networks. This approach allows the model to make fuller use of unlabeled data, thereby improving model performance with limited labeled data.

[0011] Hybrid approaches refer to strategies that combine different semi-supervised learning techniques to utilize labeled and unlabeled data from multiple perspectives. Hybrid approaches attempt to extract maximum information by fusing generative and discriminative methods, or different types of discriminative methods (such as consistency regularization and self-training), thereby improving the performance of learning algorithms. While hybrid approaches have shown great promise in other fields, their application in speech emotion recognition is currently limited.

[0012] Most semi-supervised speech emotion recognition research assumes that unlabeled and labeled emotion samples exist in the same label space, ignoring the specificity of emotion categories and potentially leading to the accumulation of errors. The feature distribution of sample data is crucial information in determining its emotion category, yet most researchers neglect the proper analysis and effective reuse of the feature distribution of labeled data. Differences in annotators' subjective perceptions, the transition between different emotion categories within the speech segments to be annotated, or variations in the strength of expression over time can result in inconsistent labels from different annotators. Summary of the Invention

[0013] To address the aforementioned issues and solve the problems of semi-supervised speech emotion recognition in unlabeled data processing, this paper analyzes the statistical relationship between evaluation clusters and emotion categories by using cluster probability scores to assess the feature distribution and patterns of the data, and alleviates the data imbalance problem by using reasonable emotion category normalization.

[0014] A semi-supervised speech emotion recognition method based on K-means clustering includes the following steps:

[0015] S1. Acquire the speech signal, process the speech signal into spectrogram data, and divide it into labeled spectrogram data and unlabeled spectrogram data.

[0016] S2. Use the EfficientNet-B1 model to extract features from the labeled spectrogram data to obtain feature T1. Perform K-means clustering on feature T1 to obtain the probability of sentiment category proportion in the cluster.

[0017] A classification model is constructed, and the model is pre-trained using labeled spectrogram data to obtain sentiment probability scores. The pre-training effect of the classification model is judged based on the proportion probability of sentiment categories in the clusters and the sentiment probability scores, and the pre-trained classification model is obtained.

[0018] The EfficientNet-B1 model, trained on labeled spectrogram data, is used to extract features from unlabeled spectrogram data to obtain feature T2.

[0019] The feature mean μ of different emotion categories is calculated using feature T1. The feature T1 is then clustered using K-means clustering with μ as the initial cluster center. The final cluster center is obtained by iteratively calculating the mean and standard deviation of the distance from feature T1 to the final cluster center. The sum of the mean and standard deviation of the distance is used as the filtering threshold.

[0020] Calculate the distance from feature T2 to each cluster, discard data whose distance exceeds a set threshold, and obtain the filtered unlabeled spectrogram data;

[0021] S3. Perform weak augmentation on labeled spectrogram data, and perform weak and strong augmentation on filtered unlabeled spectrogram data respectively. Input the augmented data into the pre-trained classification model for training, and use the loss function to constrain it to obtain the final classification model.

[0022] S4. Input the spectrogram data to be identified into the final classification model to obtain the speech emotion recognition result.

[0023] Furthermore, K-means clustering is performed on feature T1 to obtain the probability of the proportion of sentiment categories in the clusters, expressed as follows:

[0024]

[0025] in, N i N represents the total number of samples in a certain cluster category. j Let i represent the total number of samples for a certain sentiment category, i∈{1,2,…,k},j∈{1,2,…,c}, where k and c represent the number of clusters and the number of sentiment categories, respectively.

[0026] Furthermore, the pre-training effect of the classification model is judged based on the probability proportion of sentiment categories in the clusters and the sentiment probability scores. Specifically, the probability proportion of sentiment categories in the clusters and the sentiment probability scores are weighted and fused, and the pre-training effect of the classification model is judged based on the weighted fusion result.

[0027] P out =αP classify +(1-α)P cluster

[0028] Among them, P out The result is a weighted fusion, where α is the weight and P is the weight. cluster P represents the probability of the sentiment category being represented in the cluster. classify This represents the probability score for emotional state.

[0029] Further, in step S2, before using the unlabeled spectrogram data, the unlabeled spectrogram data is filtered, and the unlabeled data is manually labeled. The set of manually labeled data for the same unlabeled spectrogram data is as follows: Let S be the set of artificially labeled data for the t-th unlabeled spectrogram data, and let S be the set of labels for the labeled spectrogram data. label ,when Remove the t-th unlabeled spectrogram from the unlabeled spectrogram data.

[0030] Furthermore, the EfficientNet-B1 model trained with labeled spectrogram data is used to extract features from unlabeled spectrogram data, extracting feature T. t-SNE is then used to perform nonlinear dimensionality reduction on feature T to obtain feature T2.

[0031] Furthermore, the constraint using the loss function is specifically as follows:

[0032] The loss for training a pre-trained classification model using weakly augmented labeled spectrogram data is:

[0033]

[0034] Among them, L l H(p) is the loss used to train a pre-trained classification model by weakly augmenting labeled spectrogram data, where B is the batch size of the labeled spectrogram data, and H(p) is the loss. l ,p(y|A w (x l ))) is p l and p(y|A w (x l Cross-entropy loss between two probability distributions, p l Let p(y|A) be the pseudo-label probability distribution. w (x l )) indicates that the input to the classification model is A w (x l The predicted class probability distribution, A w (x l ) represents weakly enhanced labeled spectrogram data, x l Represents labeled spectrogram data;

[0035] The filtered unlabeled spectrogram data is weakly augmented and then input into the pre-trained classification model to obtain pseudo-labels. The filtered unlabeled spectrogram data is strongly augmented and then input into the pre-trained classification model to obtain the speech emotion recognition result A3. The speech emotion recognition result A3 is then matched with the pseudo-labels and the loss is calculated.

[0036]

[0037] Among them, L u The filtered unlabeled spectrogram data is strongly augmented and then input into the pre-trained classification model for training. The cross-entropy loss is used for matching with pseudo-labels. β is a hyperparameter, and βB is the batch size of the filtered unlabeled spectrogram data. for and p(y|A s (x u Cross-entropy loss between two probability distributions q u It is the probability distribution of the class predicted by the classification model for the filtered unlabeled spectrogram data, p(y|A s (x u )) indicates that the input to the classification model is A s (x u The predicted class probability distribution, A s (x u ) represents the strongly enhanced, filtered, unlabeled spectrogram data, x u This represents the filtered, unlabeled spectrogram data, where τ is the confidence level.

[0038] The entire algorithm model update is completed by minimizing the total loss of labeled spectrogram data and filtered unlabeled spectrogram data through backpropagation. The total loss of labeled spectrogram data and filtered unlabeled spectrogram data is:

[0039] L = L l +ωL u

[0040] Where L is the total loss of labeled spectrogram data and filtered unlabeled spectrogram data, and ω represents the weight of the filtered unlabeled spectrogram data.

[0041] This invention also proposes a semi-supervised speech emotion recognition system based on K-means clustering, comprising:

[0042] The data acquisition module is used to acquire speech signals, process the speech signals into spectrogram data, and divide them into labeled spectrogram data and unlabeled spectrogram data.

[0043] The model pre-training and data filtering module is used to extract features from labeled spectrogram data using the EfficientNet-B1 model to obtain feature T1, and to perform K-means clustering on feature T1 to obtain the probability of sentiment category proportion in the cluster.

[0044] A classification model is constructed, and the model is pre-trained using labeled spectrogram data to obtain sentiment probability scores. The pre-training effect of the classification model is judged based on the proportion probability of sentiment categories in the clusters and the sentiment probability scores, and the pre-trained classification model is obtained.

[0045] The EfficientNet-B1 model, trained on labeled spectrogram data, is used to extract features from unlabeled spectrogram data to obtain feature T2.

[0046] The feature mean μ of different emotion categories is calculated using feature T1. The feature T1 is then clustered using K-means clustering with μ as the initial cluster center. The final cluster center is obtained by iteratively calculating the mean and standard deviation of the distance from feature T1 to the final cluster center. The sum of the mean and standard deviation of the distance is used as the filtering threshold.

[0047] Calculate the distance from feature T2 to each cluster, discard data whose distance exceeds a set threshold, and obtain the filtered unlabeled spectrogram data;

[0048] The model training module is used to perform weak augmentation on labeled spectrogram data, and weak and strong augmentation on filtered unlabeled spectrogram data respectively. The augmented data are then input into the pre-trained classification model for training, and a loss function is used to constrain the model to obtain the final classification model.

[0049] The result acquisition module is used to input the spectrogram data to be identified into the final classification model to obtain the speech emotion recognition result.

[0050] The present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech emotion recognition method.

[0051] The present invention also proposes an electronic device, including a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program including computer-readable instructions, and the processor is configured to invoke the computer-readable instructions to execute the above-described voice emotion recognition method.

[0052] The beneficial effects of the technical solution provided by this invention are:

[0053] This invention uses unlabeled data for K-means clustering, and then fuses the results of the classification model with the results of the classification model using labeled data to pre-train the classification model, thus alleviating the data imbalance problem and enabling the classification model to focus more on sentiment features. It integrates K-means clustering with a semi-supervised method, using semi-supervised clustering to filter out unlabeled data, making more effective use of it. Finally, it uses the pre-trained classification model, labeled data, and filtered unlabeled data to further train the classification model, improving its performance. Attached Figure Description

[0054] Figure 1 This is a flowchart of the speech emotion recognition method according to an embodiment of the present invention;

[0055] Figure 2 This is a block diagram of an electronic device according to an exemplary embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0057] The flowchart of the speech emotion recognition method of this invention is as follows: Figure 1 Specifically, it includes the following steps:

[0058] S1. Acquire the speech signal, process the speech signal into spectrogram data, and divide it into labeled spectrogram data and unlabeled spectrogram data.

[0059] The acquired raw speech signal needs to undergo necessary data preprocessing operations, including sampling, quantization, pre-emphasis, and windowing, to generate an original sentiment spectrogram. This spectrogram is initially a long spectrogram, which is then cropped with a fixed step size to obtain a spectrogram dataset. A portion of the data in the spectrogram dataset is labeled to obtain labeled spectrogram data, while the unlabeled portion remains unlabeled.

[0060] S2. The EfficientNet-B1 model is used to extract features from the labeled spectrogram data, and the feature vectors are flattened into 1D vectors to obtain feature T1. K-means clustering is then performed on feature T1 to obtain the probability of sentiment category proportion in each cluster. Feature-level K-means clustering is less affected by the information of the underlying pixels and has better robustness.

[0061] Assuming that after K-means clustering, the proportion relationship can be expressed as:

[0062]

[0063] Where i∈{1,2,…,k}, j∈{1,2,…,c}, k and c represent the number of clusters and the number of sentiment categories, respectively. To ensure a better mapping relationship, k is generally set to c, that is, consistent with the number of sentiment categories. ij This represents the number of samples belonging to sentiment category j within cluster i. Based on fundamental statistical theory, the normalized frequencies obtained from these frequencies can be approximated as the probability of the sentiment category's proportion within the cluster, reflecting the overall sentiment characteristic distribution information of the samples in the cluster.

[0064] When there are significant differences in the number of samples for each sentiment category, directly using the above formula will result in the probability of the sentiment category being mainly influenced by the majority class. In this case, the frequency is replaced, and K-means clustering is performed on feature T1 to obtain the probability of the sentiment category in each cluster as follows:

[0065]

[0066] in, N i N represents the total number of samples in a certain cluster category. j Let represent the total number of samples in a certain sentiment category, i∈{1,2,…,k},j∈{1,2,…,c}, where k and c represent the number of clusters and the number of sentiment categories, respectively. The weight of the number of samples in that sentiment category within that cluster is used to represent the total number of sentiment categories. Replace the number n of samples belonging to the sentiment category in the cluster. ij This helps to achieve an optimal balance between preserving the original proportions and mitigating data imbalance.

[0067] A classification model is constructed. This model is a conventional classification model; commonly used classification models in machine learning include decision trees, support vector machines, random forests, gradient boosting trees, and CaiT-XXS. This embodiment of the invention uses the CaiT-XXS model. The classification model is pre-trained using labeled spectrogram data to obtain sentiment probability scores. The sentiment category proportion probability in each cluster is weighted and fused with the sentiment probability scores. The pre-training effect of the classification model is judged based on the weighted fusion result, resulting in the pre-trained classification model.

[0068] P out =αP classify +(1-α)P cluster

[0069] Among them, P out The result is a weighted fusion, where α is the weight and P is the weight. cluster P represents the probability of the sentiment category being represented in the cluster. classifyThe sentiment probability score is calculated by fusing the probability percentage of sentiment categories within each cluster with the sentiment probability score obtained from the classification model. This helps the model focus more on sentiment-related information.

[0070] Unlabeled spectrogram data was filtered, and unlabeled data were manually labeled. The set of manually labeled data for the same unlabeled spectrogram data was then defined as follows: Let S be the set of artificially labeled data for the t-th unlabeled spectrogram data, and let S be the set of labels for the labeled spectrogram data. label ,when Remove the t-th unlabeled spectrogram from the unlabeled spectrogram data.

[0071] The EfficientNet-B1 model, trained on labeled spectrogram data, extracts features from the filtered unlabeled spectrogram data and uses t-SNE to perform nonlinear dimensionality reduction on the features to obtain feature T2.

[0072] Calculate the feature mean μ for different emotion categories using feature T1, set the number of clusters to be equal to the number of emotion categories, and use the feature mean μ of different emotion categories as the initial center of each cluster. This can be represented as:

[0073]

[0074] Where, n l It is the number of all features of feature T1. It is the l-th feature in feature T1.

[0075] K-means clustering is performed on feature T1 using μ as the initial cluster center. The final cluster center is obtained by iterative processing. The mean and standard deviation of the distance from feature T1 to the cluster center are calculated, and the sum of the mean and standard deviation of the distance is used as the filtering threshold.

[0076] dist=||f-μ new ||,

[0077]

[0078] Where dist represents the distance from each category of feature T1 to the cluster center, f represents the feature of each category in feature T1, T represents the filtering threshold, and μ new Indicates the final cluster center. and These represent the mean and standard deviation of the distances from the labeled data to the cluster centers, respectively. It is the distance from the labeled data to the cluster center.

[0079] Calculate the distance from feature T2 to each cluster, discard data in feature T2 whose distance exceeds a set threshold, and obtain the filtered unlabeled spectrogram data;

[0080] S3. Perform weak augmentation on the labeled spectrogram data, and perform weak and strong augmentation on the filtered unlabeled spectrogram data respectively. Input the augmented data together into the pre-trained classification model for training, and use the loss function for constraints to obtain the final classification model.

[0081] The loss for training a pre-trained classification model using weakly augmented labeled spectrogram data is:

[0082]

[0083] Among them, L l H(p) is the loss used to train a pre-trained classification model by weakly augmenting labeled spectrogram data, where B is the batch size of the labeled spectrogram data, and H(p) is the loss. l ,p(y|A w (x l ))) is p l and p(y|A w (x l Cross-entropy loss between two probability distributions, p l Let p(y|A) be the pseudo-label probability distribution. w (x l )) indicates that the input to the classification model is A w (x l The predicted class probability distribution, A w (x l ) represents weakly enhanced labeled spectrogram data, x l This represents labeled spectrogram data.

[0084] The filtered unlabeled spectrogram data is weakly augmented and then input into the pre-trained classification model to obtain pseudo-labels. The filtered unlabeled spectrogram data is strongly augmented and then input into the pre-trained classification model to obtain the speech emotion recognition result A3. The speech emotion recognition result A3 is then matched with the pseudo-labels and the loss is calculated.

[0085]

[0086] Among them, L u The filtered unlabeled spectrogram data is strongly augmented and then input into the pre-trained classification model for training. The cross-entropy loss is used for matching with pseudo-labels. β is a hyperparameter, and βB is the batch size of the filtered unlabeled spectrogram data. When max(q) is satisfied... u When )≥τ, (max(q)u Only when )≥τ) is 1, is L correct. u The calculation does not satisfy max(q) u When )≥τ, (max(q) u )≥τ) is 0, for and p(y|A s (x u Cross-entropy loss between two probability distributions q u It is the probability distribution of the class predicted by the classification model for the filtered unlabeled spectrogram data, p(y|A s (x u )) indicates that the input to the classification model is A s (x u The predicted class probability distribution, A s (x u ) represents the strongly enhanced, filtered, unlabeled spectrogram data, x u This represents the filtered, unlabeled spectrogram data, where τ is the confidence level.

[0087] The entire algorithm model update is completed by minimizing the total loss of labeled spectrogram data and filtered unlabeled spectrogram data through backpropagation. The total loss of labeled spectrogram data and filtered unlabeled spectrogram data is:

[0088] L = L l +ωL u

[0089] Where L is the total loss of labeled spectrogram data and filtered unlabeled spectrogram data, and ω represents the weight of the filtered unlabeled spectrogram data.

[0090] S4. Input the spectrogram data to be identified into the final classification model to obtain the speech emotion recognition result.

[0091] This invention also proposes a semi-supervised speech emotion recognition system based on K-means clustering, comprising:

[0092] The data acquisition module is used to acquire speech signals, process the speech signals into spectrogram data, and divide them into labeled spectrogram data and unlabeled spectrogram data.

[0093] The model pre-training and data filtering module is used to extract features from labeled spectrogram data using the EfficientNet-B1 model to obtain feature T1, and to perform K-means clustering on feature T1 to obtain the probability of sentiment category proportion in the cluster.

[0094] A classification model is constructed, and the model is pre-trained using labeled spectrogram data to obtain sentiment probability scores. The pre-training effect of the classification model is judged based on the proportion probability of sentiment categories in the clusters and the sentiment probability scores, and the pre-trained classification model is obtained.

[0095] The EfficientNet-B1 model, trained on labeled spectrogram data, is used to extract features from unlabeled spectrogram data to obtain feature T2.

[0096] The feature mean μ of different emotion categories is calculated using feature T1. The feature T1 is then clustered using K-means clustering with μ as the initial cluster center. The final cluster center is obtained by iteratively calculating the mean and standard deviation of the distance from feature T1 to the final cluster center. The sum of the mean and standard deviation of the distance is used as the filtering threshold.

[0097] Calculate the distance from feature T2 to each cluster, discard data whose distance exceeds a set threshold, and obtain the filtered unlabeled spectrogram data;

[0098] The model training module is used to perform weak augmentation on labeled spectrogram data, and weak and strong augmentation on filtered unlabeled spectrogram data respectively. The augmented data are then input into the pre-trained classification model for training, and a loss function is used to constrain the model to obtain the final classification model.

[0099] The result acquisition module is used to input the spectrogram data to be identified into the final classification model to obtain the speech emotion recognition result.

[0100] In one exemplary embodiment, a computer-readable storage medium is included, which stores a computer program that, when executed by a processor, implements the above-described voice emotion recognition method.

[0101] Please see Figure 2 In one exemplary embodiment, the device further includes an electronic device including at least one processor, at least one memory, and at least one communication bus.

[0102] The memory stores a computer program, which includes computer-readable instructions. The processor calls the computer-readable instructions stored in the memory through the communication bus to execute the aforementioned voice emotion recognition method.

[0103] The experimental results using the method of this invention are presented below, and compared with other methods. Table 1 shows the comparison between the recognition results at each stage and the baseline results. The comparison includes the baseline model that only classifies, the classification model using the best clustering assistance, and the semi-supervised learning results of mixed unlabeled data. Compared with the model that only classifies, the clustering assistance method proposed in this invention helps to improve the model's recognition performance; therefore, the model weights obtained based on clustering assistance are selected as pre-training weights. To verify the effectiveness of the proposed filtering method, for unlabeled data, in addition to using only the filtered data, the filtered data is replaced with the filtered data while keeping the quantity unchanged, for experimental comparison. As can be seen from the table, compared with EMODB (Berlin Emotional Database), the recognition performance on both datasets is further improved when only the filtered data is used. Since the unlabeled data and IEMOCAP-4 (Interactive emotional dyadic motion capture database) have the same source and there are no differences in language or collection conditions, the improvement in recognition performance by semi-supervised learning is more significant on IEMOCAP-4 (from 67.98% to 71.11% on IEMOCAP-4, and from 89.09% to 90.40% on EMODB). The negative impact of the filtered data on model learning is also greater (71.11% on IEMOCAP-4 compared to 70.57%, and 90.40% on EMODB compared to 90.13%).

[0104] Table 1

[0105]

[0106] Table 2 compares the results of the method in this embodiment with those of other methods. The methods involved include: SVM (Support Vector Machines), MTL (Multi-task Learning), CMKT (Cross-modal Knowledge Transfer), SSL (Semi-Supervised Learning), FL (Federated Learning), MTL-AUG (Multitask Learning From Augmented Auxiliary Data), CNN (Convolutional Neural Networks), BiLSTM (Bi-directional Long Short-Term Memory), and Att (Attention). Compared with other methods, the method in this embodiment significantly improves both WA (weighted accuracy) and UA (unweighted accuracy) metrics.

[0107] Table 2

[0108]

[0109] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A semi-supervised speech emotion recognition method based on K-means clustering, characterized in that, Includes the following steps: S1. Acquire the speech signal, process the speech signal into spectrogram data, and divide it into labeled spectrogram data and unlabeled spectrogram data. S2. Use the EfficientNet-B1 model to extract features from the labeled spectrogram data to obtain feature T1. Perform K-means clustering on feature T1 to obtain the probability of sentiment category proportion in the cluster. A classification model is constructed, and the model is pre-trained using labeled spectrogram data to obtain sentiment probability scores. The pre-training effect of the classification model is judged based on the proportion probability of sentiment categories in the clusters and the sentiment probability scores, and the pre-trained classification model is obtained. The EfficientNet-B1 model, trained on labeled spectrogram data, is used to extract features from unlabeled spectrogram data to obtain feature T2. The feature mean μ of different emotion categories is calculated using feature T1. The feature T1 is then clustered using K-means clustering with μ as the initial cluster center. The final cluster center is obtained by iteratively calculating the mean and standard deviation of the distance from feature T1 to the final cluster center. The sum of the mean and standard deviation of the distance is used as the filtering threshold. Calculate the distance from feature T2 to each cluster, discard data whose distance exceeds the filtering threshold, and obtain the filtered unlabeled spectrogram data; S3. Perform weak augmentation on labeled spectrogram data, and perform weak and strong augmentation on filtered unlabeled spectrogram data respectively. Input the augmented data into the pre-trained classification model for training, and use the loss function to constrain it to obtain the final classification model. S4. Input the spectrogram data to be identified into the final classification model to obtain the speech emotion recognition result; Perform K-means clustering on feature T1 to obtain the probability of sentiment category proportion in each cluster, as follows: in, , This represents the total number of samples in a given cluster category. This represents the total number of samples for a particular emotion category. k and c represent the number of cluster categories and the number of sentiment categories, respectively; Using a loss function for constraints specifically involves: The loss for training a pre-trained classification model using weakly augmented labeled spectrogram data is: in, The loss is used to train a pre-trained classification model after weakly augmenting labeled spectrogram data. For the batch size of labeled spectrogram data, for and Cross-entropy loss between two probability distributions For pseudo-label probability distribution, The input to the classification model is The predicted class probability distribution. This represents labeled spectrogram data after weak enhancement. Represents labeled spectrogram data; The filtered unlabeled spectrogram data is weakly augmented and then input into the pre-trained classification model to obtain pseudo-labels. The filtered unlabeled spectrogram data is strongly augmented and then input into the pre-trained classification model to obtain the speech emotion recognition result A3. The speech emotion recognition result A3 is then matched with the pseudo-labels and the loss is calculated. in, The filtered, unlabeled spectrogram data is strongly augmented and then input into a pre-trained classification model for training. Cross-entropy loss is used to match the pseudo-labels. It's a hyperparameter. The batch size for the filtered, unlabeled spectrogram data. for and Cross-entropy loss between two probability distributions , It is the probability distribution of classes predicted by the classification model for the filtered, unlabeled spectrogram data. The input to the classification model is The predicted class probability distribution. This represents the strongly enhanced, filtered, unlabeled spectrogram data. This represents the filtered, unlabeled spectrogram data. It is the confidence level; The entire algorithm model update is completed by minimizing the total loss of labeled spectrogram data and filtered unlabeled spectrogram data through backpropagation. The total loss of labeled spectrogram data and filtered unlabeled spectrogram data is: Where L represents the total loss of labeled spectrogram data and filtered unlabeled spectrogram data. This represents the weight of the unlabeled spectrogram data after filtering.

2. The semi-supervised speech emotion recognition method based on K-means clustering according to claim 1, characterized in that, The pre-training performance of the classification model is judged based on the probability of sentiment category proportion in each cluster and the sentiment probability score. Specifically, the probability of sentiment category proportion in each cluster and the sentiment probability score are weighted and fused, and the pre-training performance of the classification model is judged based on the weighted fusion result. in, The result is a weighted fusion, where α is the weight. The probability of the percentage of sentiment categories in a cluster. This represents the probability score for emotional state.

3. The semi-supervised speech emotion recognition method based on K-means clustering according to claim 1, characterized in that, In step S2, before using the unlabeled spectrogram data, the unlabeled spectrogram data is filtered, and the unlabeled data is manually labeled. The set of manually labeled data for the same unlabeled spectrogram data is as follows: , Let be the set of artificial labels for the t-th unlabeled spectrogram data, and let be the set of labels for the labeled spectrogram data. ,when Remove the t-th unlabeled spectrogram from the unlabeled spectrogram data.

4. The semi-supervised speech emotion recognition method based on K-means clustering according to claim 1, characterized in that, The EfficientNet-B1 model, trained on labeled spectrogram data, extracts features from unlabeled spectrogram data, extracting feature T. t-SNE is then used to perform non-linear dimensionality reduction on feature T to obtain feature T2.

5. A semi-supervised speech emotion recognition system based on K-means clustering, characterized in that, include: The data acquisition module is used to acquire speech signals, process the speech signals into spectrogram data, and divide them into labeled spectrogram data and unlabeled spectrogram data. The model pre-training and data filtering module is used to extract features from labeled spectrogram data using the EfficientNet-B1 model to obtain feature T1, and to perform K-means clustering on feature T1 to obtain the probability of sentiment category proportion in the cluster. A classification model is constructed, and the model is pre-trained using labeled spectrogram data to obtain sentiment probability scores. The pre-training effect of the classification model is judged based on the proportion probability of sentiment categories in the clusters and the sentiment probability scores, and the pre-trained classification model is obtained. The EfficientNet-B1 model, trained on labeled spectrogram data, is used to extract features from unlabeled spectrogram data to obtain feature T2. The feature mean μ of different emotion categories is calculated using feature T1. The feature T1 is then clustered using K-means clustering with μ as the initial cluster center. The final cluster center is obtained by iteratively calculating the mean and standard deviation of the distance from feature T1 to the final cluster center. The sum of the mean and standard deviation of the distance is used as the filtering threshold. Calculate the distance from feature T2 to each cluster, discard data whose distance exceeds the filtering threshold, and obtain the filtered unlabeled spectrogram data; The model training module is used to perform weak augmentation on labeled spectrogram data, and weak and strong augmentation on filtered unlabeled spectrogram data respectively. The augmented data are then input into the pre-trained classification model for training, and a loss function is used to constrain the model to obtain the final classification model. The result acquisition module is used to input the spectrogram data to be identified into the final classification model to obtain the speech emotion recognition result; Perform K-means clustering on feature T1 to obtain the probability of sentiment category proportion in each cluster, as follows: in, , This represents the total number of samples in a given cluster category. This represents the total number of samples for a particular emotion category. k and c represent the number of cluster categories and the number of sentiment categories, respectively; Using a loss function for constraints specifically involves: The loss for training a pre-trained classification model using weakly augmented labeled spectrogram data is: in, The loss is used to train a pre-trained classification model after weakly augmenting labeled spectrogram data. For the batch size of labeled spectrogram data, for and Cross-entropy loss between two probability distributions For pseudo-label probability distribution, The input to the classification model is The predicted class probability distribution. This represents labeled spectrogram data after weak enhancement. Represents labeled spectrogram data; The filtered unlabeled spectrogram data is weakly augmented and then input into the pre-trained classification model to obtain pseudo-labels. The filtered unlabeled spectrogram data is strongly augmented and then input into the pre-trained classification model to obtain the speech emotion recognition result A3. The speech emotion recognition result A3 is then matched with the pseudo-labels and the loss is calculated. in, The filtered, unlabeled spectrogram data is strongly augmented and then input into a pre-trained classification model for training. Cross-entropy loss is used to match the pseudo-labels. It's a hyperparameter. The batch size for the filtered, unlabeled spectrogram data. for and Cross-entropy loss between two probability distributions , It is the probability distribution of classes predicted by the classification model for the filtered, unlabeled spectrogram data. The input to the classification model is The predicted class probability distribution. This represents the strongly enhanced, filtered, unlabeled spectrogram data. This represents the filtered, unlabeled spectrogram data. It is the confidence level; The entire algorithm model update is completed by minimizing the total loss of labeled spectrogram data and filtered unlabeled spectrogram data through backpropagation. The total loss of labeled spectrogram data and filtered unlabeled spectrogram data is: Where L represents the total loss of labeled spectrogram data and filtered unlabeled spectrogram data. This represents the weight of the unlabeled spectrogram data after filtering.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method as described in any one of claims 1-4.

7. An electronic device, characterized in that, The device includes a processor and a memory interconnected thereto, wherein the memory is used to store a computer program, the computer program including computer-readable instructions, and the processor is configured to invoke the computer-readable instructions to perform the method as described in any one of claims 1-4.

Citation Information

Patent Citations

  • Voice emotion recognition method based on attention CNN Bi-GRU fusion visual information

    CN113643723A

  • Voiceprint recognition model generation method, voiceprint recognition method and voiceprint recognition system

    CN116110403A