A speech emotion recognition method based on diffusion enhancement and multi-granularity feature fusion

Abstract

The present invention belongs to the field of emotional computing technology, and specifically relates to a method for speech emotion recognition based on diffusion enhancement and multi-granularity feature fusion, which converts audio into a mel-spectrogram; inputs the mel-spectrogram into a diffusion model, and then gradually performs forward diffusion to inject noise and reverse diffusion to remove noise to restore clearer audio; extracts features of the enhanced audio, and then extracts frame-level features, segment-level features, and speech-level features from the audio, and then performs adaptive feature selection, and performs deep fusion through weight position dependency; inputs the fused multi-granularity speech features into a dynamic graph convolution emotion classification model to generate an emotion state recognition result. The present invention can provide services such as multi-classification speech emotion recognition and speech quality enhancement, thereby providing a more humane response in interaction.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech information analysis, and in particular relates to a speech emotion recognition method based on diffusion enhancement and multi-granularity feature fusion. Background Art

[0002] Speech emotion recognition provides a more efficient and humanized service for the field of human-computer interaction. It builds a mapping relationship between speech information carriers and emotional dimensions, thereby giving computers the ability to recognize and understand human emotions. Its application purpose is to improve the naturalness and fluency of human-computer interaction, in order to solve problems such as insufficient understanding of user emotional states, poor personalized service experience, and low communication efficiency. Speech emotion recognition provides an efficient pattern recognition that can better understand the emotions and needs of users and stakeholders. With the continuous development of artificial intelligence and speech recognition technology, speech emotion recognition technology is increasingly widely used in many fields. It can already achieve high-accuracy emotion recognition and identify the speaker's emotional state to respond quickly. This technology has broad application prospects in education, medical care, business and other fields, which can help people better understand and communicate with each other and improve the efficiency and comfort of human-computer interaction. For example, it is used in remote teaching to monitor students' emotional states in real time and make appropriate treatments, thereby improving teaching quality; it is used to identify the emotions of mental health patients such as depression, and provide valuable reference information for treatment; it is also often used in intelligent customer service systems to improve user satisfaction. However, due to the complexity and diversity of human emotional expression, existing speech emotion recognition models often have difficulty in achieving satisfactory results when dealing with complex emotional scenes, especially when dealing with multiple categories of emotions, and emotion recognition failures may occur due to imbalanced sample categories. At the same time, when processing speech signals in noisy environments, traditional noise reduction methods such as spectral subtraction, Wiener filtering, subspace methods, and deep learning methods such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) all play an important role. However, these methods are prone to distortion of speech signals, affecting the quality of speech hearing, and cannot effectively retain the emotional information in speech signals. In addition, most existing speech emotion analysis models adopt a single-task structure and cannot simultaneously analyze the characteristics of speech signals from multiple dimensions, resulting in limited accuracy in emotion classification. Summary of the invention

[0003] The purpose of the present invention is to propose a speech emotion recognition method based on diffusion enhancement and multi-granularity feature fusion. The method first performs data enhancement on the Mel-spectrogram of the audio through a diffusion model based on stochastic differential equations, simulates noise interference in the real world and performs denoising to restore clearer audio. Then, three layers of features at the frame level, segment level, and speech level are extracted from the speech signal as coarse-grained features, which are fused with the fine-grained features selected by the mutual information incremental feature elimination adaptive mechanism and the weight position dependence mechanism to form multi-granularity features. Finally, the multi-granularity speech features are input into the graph convolutional neural network for specific emotion classification to obtain the corresponding emotional state.

[0004] The present invention is achieved through the following technical solutions.

[0005] A speech emotion recognition method based on diffusion enhancement and multi-granularity feature fusion, the steps are as follows:

[0006] Step 1: Obtain speech data and perform standardization and emotion classification to obtain an audio data set with emotion category labels;

[0007] Step 2: Logarithmically transform the audio data set into a Mel-spectrogram, and use a diffusion model based on stochastic differential equations to perform data enhancement on the Mel-spectrogram to generate enhanced audio;

[0008] Step 3: Extract the low-level descriptors of the enhanced audio and perform statistical calculations to obtain high-level descriptors as frame-level features;

[0009] Step 4: Input the frame-level features into the bidirectional long short-term memory network. The forward and reverse long short-term memory networks combine the previous and next context information at each time step to capture the dependency of features in a longer time range as segment-level features.

[0010] Step 5: Extract speech-level features of the enhanced audio;

[0011] Step 6: For the three speech features of frame-level features, segment-level features and utterance-level features, use mutual information to select speech features whose correlation with the target variable is greater than the set threshold, then use the mutual information between speech features to remove redundant speech features, and recursively eliminate the selected speech features to retain the final speech features;

[0012] Step 7: Input the final speech features into the multi-head attention mechanism, learn the relationship between speech features from different angles through multiple attention heads, dynamically assign weights, and use the cross-attention mechanism to fuse different types of speech features to obtain multi-granularity speech features;

[0013] Step 8: Input the multi-granularity speech features into the dynamic graph convolutional sentiment classification model for speech emotion recognition and output the emotion category.

[0014] Further preferably, the data enhancement described in step 2 is divided into two processes: forward diffusion denoising and reverse diffusion denoising.

[0015] Further preferably, in step three, the Opensmile tool is used directly on the enhanced audio to extract low-level descriptors of the audio frame by frame, and the mean and standard deviation statistics are calculated and aggregated to obtain high-level descriptors as frame-level features.

[0016] Further preferably, in step 4, the bidirectional long short-term memory network has H hidden units, and the dimension is expressed as , for each time step t, the bidirectional long short-term memory network outputs a 2H-dimensional vector, and the output result is:

[0017] ;

[0018] For each segment, the features output by the bidirectional long short-term memory network are averaged at each time step to obtain segment-level features:

[0019] ;

[0020] Among them, h segment is the segment-level feature, h t is the feature of the time step t output by the bidirectional long short-term memory network, and T is the time step.

[0021] Further preferably, in step five, the enhanced audio is input into a Hubert model, and features are extracted using the Hubert model to form speech-level features.

[0022] Further preferably, in step six, a judgment is made based on the size of the mutual information value between the speech features. If the mutual information is greater than a set threshold, it indicates that there is a redundant relationship between the speech features and one of the speech features can be selected to be removed. After mutual information screening and redundant speech feature removal, a recursive feature elimination method is used to evaluate the contribution of each speech feature, and the speech features with the smallest contribution are gradually deleted until an optimal speech feature subset is finally obtained.

[0023] Further preferably, the step seven sets a position weight dependency mechanism, assigns different weights to the selected speech features according to multi-head attention, and uses cross attention for further fusion to obtain the final multi-granularity speech feature set.

[0024] Further preferably, in step eight, the multi-granularity speech feature set will be converted into a graph structure through a graph convolutional network, the node is a single speech feature, and the edge is defined based on the category adjacency matrix; after the graph convolution operation, the graph structure updates the node features according to the information of the adjacent nodes, uses the graph nodes for feature propagation, dynamically aggregates domain nodes to update the feature representation of the node, and then performs multi-layer convolution and inputs into the fully connected layer, and the final emotion category is output by the softmax function of the emotion classification part.

[0025] The present invention has the following advantages:

[0026] 1. The present invention can more effectively learn the emotional characteristics of speech samples of varying quality, optimize data utilization to adapt to the general situation of imbalanced emotional categories, expand from the basic four categories of emotions to more detailed emotional states, meet different application requirements, and effectively avoid overfitting or underfitting problems of general models.

[0027] 2. We use the diffusion model to enhance speech data and introduce a “predictor-corrector” sampling method. We use the Mel-spectrogram of clean speech as a reference and let the model learn how to restore a clear output close to the reference from noisy input to improve the performance of speech enhancement.

[0028] 3. Combine speech acoustic features with deep representation to make the speech emotion recognition model more adaptable to processing tasks in complex speech emotion environments, avoiding misjudgment or performance degradation when facing changing emotional states, which affects the application effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of the present invention;

[0030] FIG2 is a framework diagram of a multi-granularity feature extraction process;

[0031] Figure 3 It is a line-bar composite graph of the mean change in weighted accuracy and the mean change in F1 value for 8 emotion types under different emotional speech data sets. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0033] Reference Figure 1 , a speech emotion recognition method based on diffusion enhancement and multi-granularity feature fusion, the steps are as follows:

[0034] Step 1: Obtain speech data and perform standardization and emotion classification to obtain an audio data set with emotion category labels;

[0035] Step 2: Logarithmically transform the audio data set into a Mel-spectrogram, and use a diffusion model based on stochastic differential equations to perform data enhancement on the Mel-spectrogram to generate enhanced audio;

[0036] Step 3: Use the Opensmile tool to extract low-level descriptors of the enhanced audio frame by frame, and perform statistical calculations to obtain high-level descriptors as frame-level features;

[0037] Step 4: Input the frame-level features into the bidirectional long short-term memory network. The forward and reverse long short-term memory networks combine the previous and next context information at each time step to capture the dependency of features in a longer time range as segment-level features.

[0038] Step 5: Use the Hubert model to extract speech-level features of the enhanced audio;

[0039] Step 6: For the three speech features of frame-level features, segment-level features and utterance-level features, use mutual information to select speech features whose correlation with the target variable is greater than the set threshold, then use the mutual information between speech features to remove redundant speech features, and recursively eliminate the selected speech features to retain the final speech features;

[0040] Step 7: Input the final speech features into the multi-head attention mechanism, learn the relationship between speech features from different angles through multiple attention heads, dynamically assign weights, and use the cross-attention mechanism to fuse different types of speech features to obtain multi-granularity speech features;

[0041] Step 8: Input the multi-granularity speech features into the dynamic graph convolutional sentiment classification model for speech emotion recognition and output the emotion category.

[0042] In step 1, the speech data is divided and normalized to eliminate numerical instability caused by features of different sizes in the calculation process, prevent model overfitting, identify speech signals and perform standardization, and then perform emotion classification to obtain an audio data set with emotion category labels. The sound frequency range that a normal human ear can hear is 20 Hz to 20,000 Hz. In the present invention, in order to standardize the data, the input audio is uniformly resampled to 1,600 Hz.

[0043] In step 2, the audio data set obtained in step 1 is logarithmically transformed into a Mel-spectrogram, and then the Mel-spectrogram is diffused and converted back into audio with higher clarity. The diffusion model is implemented based on stochastic differential equations, and the sampling of stochastic differential equations uses the "predictor-corrector" method. The "predictor" uses the Euler Maruyama method to solve the stochastic differential equation to obtain the signal state at the next moment. The "corrector" uses the Markov chain Monte Carlo method to correct the predicted signal state.

[0044] The data enhancement described in step 2 is divided into two processes: forward diffusion denoising and reverse diffusion denoising. When performing audio data enhancement, it is usually necessary to process the entire audio to identify and correct the target noise and distortion parts. When selecting audio enhancement technology, the effect of the model needs to be considered. The computing resources and time consumed by the selection will increase significantly with the increase in model complexity, which greatly reduces the processing efficiency and affects the overall performance. Therefore, this embodiment chooses to use a diffusion model based on stochastic differential equations to perform audio data enhancement. The Mel-spectrogram is used as a time-frequency representation method to represent the audio signal by simulating the perceptual sensitivity of the human auditory system to sounds of different frequencies. The Mel-spectrogram of clean speech is used as a reference to allow the model to learn how to restore a clear output close to the reference from a noisy input. Through the above processing, the diffusion model based on stochastic differential equations can effectively enhance speech signals.

[0045] By gradually adding noise to the Mel spectrum graph, a forward diffusion process is performed, and the original Mel spectrum graph is taken as the initial signal. , and then add noise in each time step until it eventually becomes white noise, that is, Gaussian distribution. The forward stochastic differential equation process used in this embodiment, that is, the process from data distribution to Gaussian noise distribution, can be expressed as:

[0046] ;

[0047] in, is the signal at time step t, Represents the noise attenuation factor accumulated from time step 1 to time step t. Represents the random noise of standard normal distribution, which is responsible for gradually destroying the signal structure in the forward diffusion, making the initial signal Transition to Gaussian distribution, represents a Gaussian distribution, Represents the identity matrix. As the time step t increases, the noise attenuation factor decreases, and the initial signal Step by step random noise Instead, the signal eventually becomes a white noise sampled from a Gaussian distribution. is the key to introduce randomness, which ensures that the generation of samples is gradual during the forward diffusion process, so that the model can gradually restore the samples through the reverse process. The reverse denoising process can be expressed as:

[0048] ;

[0049] in represents the score function of the prediction noise, from x t The goal is to extract the noise from x t Removed; represents the signal at time step t-1; The scaling factor representing the noise intensity is used to control the amplitude of noise injection and is adjusted as the time step t increases to ensure that the generated samples do not completely lose the characteristics of the original data during the denoising process; z is from the standard normal distribution The noise sampled in , which introduces randomness, ensures that the denoising process is not completely deterministic. To begin, use the score function of the noise estimate Remove noise. This process is iterated in multiple time steps (from t to 0) to finally restore the original signal .

[0050] The diffusion model learns the distribution of the target audio data through two processes: forward diffusion and reverse diffusion. It does not directly model the data distribution, but achieves the purpose of generation by learning the score function of the data. The score function is the logarithmic gradient of the data distribution, which indicates how to adjust along the gradient direction of the data distribution. By learning this score function, the diffusion model can learn how to recover data from noise step by step in the reverse diffusion process. In this invention, a time-dependent score function is trained. To estimate the distribution of the target data. The specific approach is to optimize the parameter θ of the score function so that the predicted noise is close to the real noise. By minimizing the initial signal x at all time steps t 0 With Gaussian distribution x t The squared error loss function between can make the model output as close to the target value as possible. Its formula is:

[0051] ;

[0052] ;

[0053] In the formula, Represents the optimization parameter of the score function; argmin represents the minimization operation of the parameter θ; Represents the expected value operation on time step t, that is, the average loss at each moment; is a positive weight function used to balance the loss contribution at different time steps to prevent the model from focusing too much on certain time steps during training; represents the target gradient, describing the gradient from the diffused state p t Return to the initial state p 0 The conditional probability distribution of , that is, the probability density function; represents the square of the L2 norm, which is used to measure the difference between the target and the prediction. Because the probability density function can be simplified to known gradient information in the case of Gaussian distribution, it can be calculated based on the conversion function Directly calculate the target gradient. The probability density function of the Gaussian distribution has the following form:

[0054] ;

[0055] in, is the mean, is the variance, I represents the identity matrix, indicating that the noise is uniform in each dimension. In this case, the mean and variance can be obtained directly. Using the known mean and variance to calculate the log-likelihood gradient, the target gradient can be calculated and optimized, and the training of the score function can be achieved through the standard back-propagation process, which is computationally simpler.

[0056] Steps three to five are the coarse-grained feature extraction stage. The speech signal features are very rich. If a single acoustic feature is extracted, the emotion recognition effect is often unsatisfactory. In order to extract the acoustic features and deep representation of the speech signal and retain its original information on the emotional state, the present invention hierarchically uses three different speech features: frame-level features, segment-level features, and speech-level features to obtain multi-granular emotion representation information. Frame-level features are features extracted from audio within a 25-millisecond time window in units of 1 frame, and segment-level features are also features extracted in units of frames. However, unlike frame-level features, segment-level features are obtained by statistical aggregation of frame-level features in units of 100 frames, while speech-level features are features extracted in units of audio for the entire time period.

[0057] The sampling rate of the enhanced audio is still 1600 Hz. In step 3, the Opensmile tool is used directly on the enhanced audio to extract the low-level audio descriptors frame by frame, and the mean and standard deviation statistics are calculated and aggregated to obtain high-level descriptors for use as frame-level features. The frame-level feature matrix U frame =[u 1 ,u 2 ,…,u K ], where u k Represents the feature vector of the kth frame, k∈{1,2,…,K}, and K is the total number of frames.

[0058] In step 4, the frame-level features are input into the bidirectional long short-term memory network. The forward and reverse long short-term memory networks combine the previous and next context information at each time step to capture the dependency of features in a longer time range. The bidirectional long short-term memory network has H hidden units, and the dimension is expressed as , for each time step t, the bidirectional long short-term memory network outputs a 2H-dimensional ( ), the output is:

[0059] ;

[0060] For each segment, the features output by the bidirectional long short-term memory network are averaged at each time step to obtain segment-level features:

[0061] ;

[0062] Among them, h segment is the segment-level feature, h t is the feature of the time step t output by the bidirectional long short-term memory network, and T is the time step.

[0063] In step 5, the enhanced audio is input into the Hubert model, and the Hubert model is used to extract features to form speech-level features. First, the audio needs to be converted into a Mel-spectrogram, and the Mel-spectrogram features of each frame are converted into a high-dimensional representation through the Hubert model;

[0064] ;

[0065] in, represents the high-dimensional representation of the audio of the t-th frame, is the feature vector of the tth frame, which contains the deep speech representation of the audio at that time point; finally, the feature vector of each frame extracted from the Hubert model An average pooling aggregation operation is performed to form the speech-level features of the entire audio segment.

[0066] Steps 6 and 7 are the fine-grained feature fusion stage. The frame-level features, segment-level features, and speech-level features determine a unique feature set through the mutual information and incremental feature elimination algorithm and the position weight dependency mechanism. The mutual information and incremental feature elimination algorithm will "select the best" for the extracted speech features (frame-level features, segment-level features, and speech-level features) to filter out redundant features and enhance the relevance of the emotion recognition results. Step 7 sets the position weight dependency mechanism, assigns different weights to the selected speech features according to multi-head attention, and uses cross attention for further fusion to obtain the final multi-granular speech feature set. Feature selection is performed on the three-level features to retain key features for fusion to achieve better emotion recognition results.

[0067] The features extracted from multiple scales may contain redundant and irrelevant parts. Feature selection of speech features can improve the performance and computational efficiency of the model, remove redundant, irrelevant or noise features, and thus enhance the generalization ability of the model. Since speech features are associated with their emotion types, by calculating the mutual information between each speech feature and the target variable, it is possible to screen features with a strong relationship with the target variable, and by calculating the mutual information between speech features, it is also possible to remove those highly correlated speech feature pairs. The present invention makes a judgment based on the value of the mutual information between speech features. If the mutual information is greater than a set threshold, it indicates that there is a redundant relationship between the speech features, and one of the speech features can be selected to be removed. After mutual information screening and redundant speech feature removal, a recursive feature elimination method is used to evaluate the contribution of each speech feature, and the speech features with the smallest contribution are gradually deleted until an optimal speech feature subset is finally obtained.

[0068] Providing high-quality speech fusion features is the prerequisite for speech emotion recognition. The present invention designs a position weight dependency mechanism, as Figure 2 As shown in the figure, the speech features after feature selection are input into the multi-head attention mechanism. Multiple attention heads can capture the relationship between speech features from different angles, dynamically assign weights to different speech features, highlight important speech features and suppress redundant information. The cross-attention mechanism is used to fuse different types of speech features to obtain the final multi-granular speech features. In this process, each attention head learns the relationship between speech features in parallel and fuses the outputs of each head by concatenation or weighted summation to capture the relationship between speech features in different subspaces. Specifically, the input speech features are first mapped to query (Q), key (K) and value (V), and the dot product between the query and the key is calculated to obtain the attention weight. The final attention weight is obtained by weighted summing the values ​​with these weights. For each head, the corresponding attention weight is calculated independently. Finally, the outputs of all heads are weighted summed and passed through a linear layer to generate the final output. Unlike the self-attention mechanism, the query and key of the cross-attention mechanism come from different feature sources. In the present invention, speech features are used as queries and the embedding of emotional tags is used as keys. The cross-attention mechanism dynamically adjusts the weights by using different types of speech features in the calculation of queries and keys, thereby effectively fusing these heterogeneous information. According to the above method, the multi-granularity speech features finally obtained not only contain the independent information of each speech feature, but also fuse the relationship between speech features from different sources, which can more effectively express the emotional information of speech signals.

[0069] Multi-granularity speech features are a multi-level speech representation with different time ranges, which can capture both fine-grained and coarse-grained information. Frame-level features contain local time information, segment-level features contain long-term dependency information, and discourse-level features contain global semantic information. Step eight is the sentiment classification stage. First, a dynamic graph is constructed. Each node of the dynamic graph represents a different multi-granularity speech feature, and the weight of the edge represents the similarity between nodes. Because the node matrix is ​​responsible for updating the features of the node and the adjacency matrix is ​​responsible for aggregating the features of neighboring nodes, it is necessary to construct the node matrix and the adjacency matrix separately. When constructing the adjacency matrix, the similarity between nodes is first calculated, and then the self-attention mechanism is used to dynamically calculate the weighted similarity between speech features at each level (frame level, segment level, discourse level), so as to fuse the feature information at different levels and output the similarity matrix. Based on the similarity matrix, a fully connected weighted graph is constructed. The fully connected weighted graph is sparsely processed by K-nearest neighbor to obtain a sparse graph to retain the adjacency relationship useful to the model. The sparse graph is converted into an adjacency matrix of the required graph structure, and the multi-granularity speech features are constructed as the initial node feature matrix; then feature aggregation is performed in the graph convolutional network (GCN), and each layer of the graph convolutional network aggregates the features of neighboring nodes according to the adjacency matrix to obtain a new node feature matrix representation; then the node features and multi-granularity speech features are updated through the back propagation of the multi-layer graph convolutional network, realizing the function of dynamically aggregating neighboring nodes to update the feature representation of this node. In this way, each row in the node feature matrix will be updated layer by layer, gradually integrating its own features and the information of neighboring nodes, and finally obtaining a high-level feature representation of each node. The final feature representation of each node is then input into the fully connected layer, and the softmax function of the sentiment classification part outputs the final sentiment category.

[0070] In the above process, calculating the similarity matrix required for the adjacency matrix is ​​a key step, which can weightedly fuse the relationship between language features at different levels (frame level, segment level, and discourse level). In the present invention, a hierarchical similarity fusion strategy is designed, which is divided into intra-level similarity calculation and cross-level weighted similarity calculation. First, the similarity between features is calculated at each level, namely, frame level, segment level, and discourse level, to capture the local structural information at the same level. The cosine similarity is used to calculate the similarity matrix between each frame feature and other frame features to generate a frame-level similarity matrix. Q frame , segment-level similarity matrix Q segment and the utterance-level similarity matrix Q utterance , and then use the self-attention mechanism to dynamically calculate the weighted coefficients of each level of similarity matrix to obtain the final weighted similarity matrix B f , the formula is as follows:

[0071] ;

[0072] in, K frame , K segment , K utterance Respectively represent Q frame , Q segment , Q utterance The corresponding key matrix; V frame , V segment , V utterance Respectively represent Q frame , Q segment , Q utterance The corresponding value matrix; d represents the dimension of the feature; softmax is the activation function used for normalization, which is used to obtain the attention weight of each node so that the sum of the attention weight of each node is 1. This formula can be used to weight the attention weights on the value matrix to obtain the final weighted similarity matrix, which can work together with the node matrix to better construct a dynamic graph that integrates information at all levels.

[0073] The speech emotion recognition method based on diffusion enhancement and multi-granularity feature fusion of the present invention is encapsulated as a speech emotion recognition model, and the performance of the model is tested as follows.

[0074] The emotion recognition results are evaluated using three indicators: weighted accuracy, unweighted accuracy, and weighted average F1. The weighted accuracy takes into account the imbalance of categories and is suitable for cases where the categories in the data set are unevenly distributed. The unweighted accuracy provides a simple estimate of the overall accuracy, but may be dominated by the performance of the majority class. The weighted average F1 combines precision and recall, and takes into account the impact of the number of category samples. It is a comprehensive indicator for measuring the overall performance of the model.

[0075] In multi-classification problems, the problem of imbalanced sample categories in data sets is very common and important. As shown in Table 1, the number of samples in the "frustrated" emotion category of the interactive emotional binary motion capture database is much larger than that in other emotion categories. This imbalance will cause the model to often find it difficult to correctly identify the minority class, causing the model performance to degrade. Although the accuracy rate can determine the overall accuracy rate, it cannot be used as a good indicator to measure the results in the case of sample imbalance. It will result in a high accuracy rate for the majority class and a low accuracy rate for the minority class. This means that no matter what the input is, the model output tends to be the label corresponding to the majority class sample, and continued training will not alleviate this situation. If no processing is performed at this time, the model cannot be trained. There are two ways to process unbalanced data sets at the data level: undersampling and oversampling. When the difference in the ratio of positive and negative samples is large, undersampling will reduce the number of samples by a lot, so it is not considered. Oversampling will increase the number of minority class samples, so the present invention uses focal loss to perform sample balance processing, and the loss center of gravity is placed on the minority class samples through a dynamic scaling factor. The experimental results after processing are shown in Table 1. Although the weighted accuracy is slightly higher, the difference is not large, which shows that the model has a relatively consistent performance when dealing with categories of different sizes. At the same time, both the weighted and unweighted accuracy rates have reached a high level on both datasets, which shows that the model has good classification performance as a whole.

[0076] Table 1

[0077]

[0078] To further compare the performance of the models when dealing with specific emotion categories, Figure 3 The weighted accuracy and weighted F1 values ​​of eight emotion types are displayed, including neutral, depressed, excited, angry, sad, fearful, disgusted, and surprised. As can be seen from the figure, the weighted accuracy and weighted F1 values ​​of all emotion categories are high, indicating that the model performs well overall in the emotion recognition task.

[0079] In order to verify the effectiveness of the proposed method in speech emotion recognition, two ablation experiments were designed, namely speech data enhancement and multi-granularity feature fusion. In the ablation experiment of speech data enhancement, four evaluation indicators are used: perceptual evaluation quality (PESQ), extended short-time objective intelligibility (ESTOI), signal-to-interference ratio (SI-SDR) and signal-to-distortion ratio (SI-SAR). PESQ is used to measure the subjective auditory quality between the processed speech signal and the original clean speech. ESTOI is used to evaluate the intelligibility of the processed speech signal, especially in the presence of background noise. SI-SDR is used to calculate the signal-to-noise ratio between the enhanced speech signal and the original clean speech, reflecting the denoising effect. SI-SAR is used to evaluate the degree of distortion of the enhanced speech signal relative to the original clean speech. The results of the ablation experiment are shown in Table 2, from which it can be seen that the various indicators of the proposed method are better than the other three ablation schemes, reflecting the effectiveness of the proposed method. In the ablation experiment of multi-granularity feature fusion, special attention is paid to the ablation experiments conducted on features of different scales. Frame-level features mainly analyze emotional feature information based on inter-frame changes and capture fine-grained temporal dynamics. Segment-level features expand the interception time based on frame-level features to capture mid-term speech features. In contrast, speech-level features capture the global information of the speech sequence by eliminating distracting information such as sound patterns and irrelevant semantics. This section uses four indicators for evaluation: weighted accuracy, unweighted accuracy, macro F1 value, and weighted F1 value. The present invention adopts four ablation schemes, and the experimental results are shown in Table 3. The multi-granularity feature fusion method adopted by the present invention performs best in all indicators, indicating that the performance of speech emotion recognition can be significantly improved by combining and supplementing different types of features.

[0080] Table 2

[0081]

[0082] Table 3

[0083]

[0084] This embodiment provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the speech emotion recognition method based on diffusion enhancement and multi-granularity feature fusion is implemented.

[0085] The invention described above only expresses the implementation methods of the embodiments of the present invention, and cannot be understood as limiting the scope of the invention patent, nor does it impose any form of limitation on the structure of the embodiments of the present invention. It should be pointed out that for ordinary technicians in this field, several changes and improvements can be made without departing from the concept of the embodiments of the present invention, which all belong to the protection scope of the embodiments of the present invention.

Claims

1. A speech emotion recognition method based on diffusion enhancement and multi-granularity feature fusion, characterized in that: Here are the steps: Step 1: Obtain speech data and perform standardization and emotion classification to obtain an audio data set with emotion category labels; Step 2: Logarithmically transform the audio data set into a Mel-spectrogram, and use a diffusion model based on stochastic differential equations to perform data enhancement on the Mel-spectrogram to generate enhanced audio; Step 3: Use the Opensmile tool to directly extract low-level audio descriptors frame by frame on the enhanced audio, and perform statistical calculations of the mean and standard deviation to obtain high-level descriptors as frame-level features; Step 4: Input the frame-level features into the bidirectional long short-term memory network. The forward and reverse long short-term memory networks combine the previous and next context information at each time step to capture the dependency of features in a longer time range as segment-level features. Step 5: Input the enhanced audio into the Hubert model and use the Hubert model to extract features to form speech-level features; Step 6: For the three speech features of frame-level features, segment-level features and utterance-level features, use mutual information to select speech features whose correlation with the target variable is greater than the set threshold, then use the mutual information between speech features to remove redundant speech features, and recursively eliminate the selected speech features to retain the final speech features; Step 7: Input the final speech features into the multi-head attention mechanism, learn the relationship between speech features from different angles through multiple attention heads, dynamically assign weights, and use the cross-attention mechanism to fuse different types of speech features to obtain multi-granularity speech features; Step 8: Input the multi-granularity speech features into the dynamic graph convolutional sentiment classification model for speech emotion recognition and output the emotion category.

2. The speech emotion recognition method based on diffusion enhancement and multi-granularity feature fusion according to claim 1 is characterized in that: The data enhancement described in step 2 is divided into two processes: forward diffusion denoising and reverse diffusion denoising.

3. The speech emotion recognition method based on diffusion enhancement and multi-granularity feature fusion according to claim 1 is characterized in that: In step 4, the bidirectional long short-term memory network has H hidden units, and the dimension is expressed as , for each time step t, the bidirectional long short-term memory network outputs a 2H-dimensional vector, and the output result is: ； For each segment, the features output by the bidirectional long short-term memory network are averaged at each time step to obtain segment-level features: ； Among them, h segment is the segment-level feature, h t is the feature of the time step t output by the bidirectional long short-term memory network, and T is the time step.

4. The speech emotion recognition method based on diffusion enhancement and multi-granularity feature fusion according to claim 1 is characterized in that: In step 6, a judgment is made based on the mutual information value between the speech features. If the mutual information is greater than a set threshold, it indicates that there is a redundant relationship between the speech features, and one of the speech features is selected to be removed. After mutual information screening and redundant speech feature removal, the recursive feature elimination method is used to evaluate the contribution of each speech feature, and the speech features with the smallest contribution are gradually deleted until an optimal speech feature subset is finally obtained.

5. The speech emotion recognition method based on diffusion enhancement and multi-granularity feature fusion according to claim 4 is characterized in that: The step seven sets a position weight dependency mechanism, assigns different weights to the selected speech features according to multi-head attention, and uses cross attention for further fusion to obtain the final multi-granularity speech feature set.

6. The speech emotion recognition method based on diffusion enhancement and multi-granularity feature fusion according to claim 5 is characterized in that: In step eight, a dynamic graph is first constructed. Each node of the dynamic graph represents a different multi-granularity speech feature, and the weight of the edge represents the similarity between the nodes. The node matrix is ​​responsible for updating the features of the nodes, and the adjacency matrix is ​​responsible for aggregating the features of the neighboring nodes. When constructing the adjacency matrix, the similarity between the nodes is first calculated, and then the self-attention mechanism is used to dynamically calculate the weighted similarity between the speech features at each level, so as to fuse the feature information at different levels to output the similarity matrix, and construct a fully connected weighted graph based on the similarity matrix. The fully connected weighted graph is subjected to K-nearest neighbor sparsification to obtain a sparse graph, and the sparse graph is used as the adjacency matrix of the required graph structure, and the multi-granularity speech features are constructed as the initial node feature matrix. Feature aggregation is then performed in the graph convolution network, and each layer of the graph convolution network aggregates the features of the neighboring nodes according to the adjacency matrix to obtain a new node feature matrix representation. The node features are then updated through the back propagation of the multi-layer graph convolution network to realize the function of dynamically aggregating neighboring nodes to update the feature representation of the node. Each row in the node feature matrix will be updated layer by layer, and its own features and the information of neighboring nodes will be gradually integrated to finally obtain the final feature representation of each node. The final feature representation of each node is then input into the fully connected layer, and the softmax function of the sentiment classification part outputs the final sentiment category.

Images