Voice emotion recognition method and system

Abstract

The invention relates to a voice emotion recognition method and system, and belongs to the technical field of intelligent interaction. Comprises: acquiring an original voice signal; calculating a frame-level score of the original voice signal, and dividing the original voice signal into a high-reliability frame signal and a low-reliability frame signal according to the frame-level score; performing feature extraction on the high-reliability frame signal, and determining Mel-frequency cepstrum features; performing feature extraction on the low-reliability frame signal to determine rhythmic features; performing feature fusion on the Mel-frequency cepstrum features and the rhythm features to obtain target features; constructing a voice emotion recognition model; and inputting the target feature into a voice emotion recognition model, and determining an emotion category. According to the invention, the accurate judgment of the emotion is realized, and the accuracy of voice emotion recognition and the stability in a complex environment are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent interaction, and particularly relates to a speech emotion recognition method and system. BACKGROUND

[0002] The speech emotion recognition method is a kind of computing method for automatically recognizing and classifying the emotional state of a speaker from human speech signals by using signal processing, machine learning and deep learning technologies. This method usually extracts acoustic features of speech, uses classification algorithms or neural networks to identify emotions, thereby realizing objective recognition and analysis of the psychological state of the speaker.

[0003] Emotion is an indispensable information dimension in human communication, and as the most natural and convenient communication medium, speech can carry rich emotional clues. An effective speech emotion recognition method can enhance the naturalness and intelligence of human-computer interaction, making intelligent customer service, voice assistants, educational companions and other applications more emotionally intelligent. It can also be used in mental health monitoring, driving safety warning and public safety fields.

[0004] However, traditional methods usually extract a single feature from the entire speech, ignore noise frames, and lose prosodic information related to emotion. Moreover, a single feature may not capture the global information of speech emotion, leading to a decline in emotion recognition performance. When performing emotion recognition, it is difficult to fully identify the details in the speech signal through a simple convolutional neural network, resulting in the omission of key emotional information. SUMMARY

[0005] In view of the shortcomings of the prior art, the purpose of the application is to provide a speech emotion recognition method and system that can accurately determine emotions and significantly improve the accuracy of speech emotion recognition and stability in complex environments.

[0006] In a first aspect of the application, a speech emotion recognition method is provided, the method comprising:

[0007] S1, obtaining an original speech signal;

[0008] S2, calculating frame-level scores of the original speech signal, and dividing the original speech signal into high-reliability frame signals and low-reliability frame signals according to the frame-level scores;

[0009] S3, performing feature extraction on the high-reliability frame signals to determine mel-frequency cepstral features;

[0010] S4, performing feature extraction on the low-reliability frame signals to determine prosodic features;

[0011] S5, performing feature fusion on the mel-frequency cepstral features and the prosodic features to obtain target features;

[0012] S6, constructing a speech emotion recognition model, wherein the speech emotion recognition model specifically comprises: a local feature learning block, a time sequence learning module, a full connection module, and a classification module;

[0013] S7, inputting the target feature into the speech emotion recognition model to determine an emotion category.

[0014] Further, the S2 specifically comprises:

[0015] S201, calculating spectral entropy, spectral flatness, and harmonic of the original speech signal;

[0016] S202, performing weighted summation on the spectral entropy, the spectral flatness, and the harmonic to obtain a frame-level score;

[0017] S203, dividing the original speech signal with the frame-level score greater than a preset score into the high-reliability frame signal, and dividing the original speech signal with the frame-level score less than the preset score into the low-reliability frame signal.

[0018] Further, the S3 specifically comprises:

[0019] S301, performing fast Fourier transform on the high-reliability frame signal to obtain a frequency spectrum;

[0020] S302, performing Mel filtering processing on the frequency spectrum to obtain a log energy;

[0021] S303, performing discrete cosine transform on the log energy to obtain a static cepstral feature;

[0022] S304, performing dynamic feature calculation on the static cepstral feature to obtain a first-order difference feature and a second-order difference feature;

[0023] S305, splicing the static cepstral feature, the first-order difference feature, and the second-order difference feature to obtain the Mel frequency cepstral feature.

[0024] Further, the prosodic feature specifically comprises: a zero-crossing rate, a Teager energy operator, and a harmonic-to-noise ratio.

[0025] Further, the S5 specifically comprises:

[0026] S501, normalizing the Mel frequency cepstral feature and the prosodic feature;

[0027] S502, performing weighted pooling on the normalized Mel frequency cepstral feature and the prosodic feature respectively to obtain a pooled Mel frequency cepstral feature and a pooled prosodic feature;

[0028] S503, fusing the pooled mel-frequency cepstral feature and the pooled prosody feature to obtain the target feature.

[0029] Further, the local feature learning block comprises a plurality of LFLB units, each of which comprises a convolution layer, a batch normalization layer, an ELU activation function and a max-pooling layer.

[0030] Further, the S7 specifically comprises:

[0031] S701, performing local feature extraction on the target feature in the local feature learning block to obtain a sampling feature map;

[0032] S702, performing global modeling on the sampling feature map in the time series learning module to obtain an aggregated time series feature;

[0033] S703, mapping the aggregated time series feature to an emotion category space in the fully connected module to obtain a category score vector;

[0034] S704, converting the category score vector to an emotion category probability distribution in the classification module to determine the emotion category.

[0035] Further, the S701 specifically comprises:

[0036] S7011, performing convolution operation on the target feature in the convolution layer of the local feature learning block to generate a convolution feature map;

[0037] S7012, performing normalization on the convolution feature map in the batch normalization layer of the local feature learning block to obtain a normalized feature;

[0038] S7013, performing nonlinear enhancement on the normalized feature in the ELU activation function of the local feature learning block to obtain a nonlinear feature;

[0039] S7014, performing dimension reduction on the nonlinear feature in the max-pooling layer of the local feature learning block to obtain the sampling feature map.

[0040] Further, the time series learning module specifically comprises a dimension projection unit, an improved LSTM unit and a global aggregation unit;

[0041] The S702 specifically comprises:

[0042] S7021, mapping the sampling feature map to a target dimension in the dimension projection unit to obtain a mapped feature map;

[0043] S7022, In the improved LSTM unit, global dependency modeling is performed on the mapped feature map to obtain temporal features;

[0044] S7023, In the global aggregation unit, the time series features are aggregated globally to obtain the aggregated time series features.

[0045] In a second aspect, the present invention provides a voice emotion recognition system, comprising: a memory and a processor;

[0046] The memory stores an application program adapted to be executed by the processor to implement the voice emotion recognition method described in the first aspect.

[0047] The beneficial effects of this invention are as follows:

[0048] In this embodiment of the invention, the original speech signal is divided into frame-level reliability segments. Different feature extraction strategies can be adopted for speech information with different reliability levels, ensuring the accuracy of the spectral features of high-reliability frames while making full use of the prosodic information of low-reliability frames to avoid the loss of emotional information. By extracting features from high-reliability and low-reliability frame signals respectively, the different information characteristics of speech signals can be fully utilized, improving the accuracy and robustness of feature expression. The extracted features are spliced ​​and fused to unify the scale of different features, reduce redundant information interference, and avoid information imbalance. Through the local feature learning block, temporal learning module, fully connected module, and classification module of the speech emotion recognition model, short-term transient emotions and long-term gradual emotions in speech are fully captured, the probability distribution of emotion categories is determined, and accurate emotion determination is achieved, significantly improving the accuracy of speech emotion recognition and its stability in complex environments. Attached Figure Description

[0049] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0050] Figure 1 This is a flowchart illustrating a speech emotion recognition method provided in an embodiment of the present invention.

[0051] Figure 2 This is a schematic diagram of the structure of a voice emotion recognition system provided in an embodiment of the present invention. Detailed Implementation

[0052] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0053] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0054] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0055] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0056] This invention proposes a speech emotion recognition method and system to address the technical problems of traditional methods that typically extract a single feature from the entire speech segment, ignoring noise frames and losing emotion-related prosodic information. Furthermore, a single feature may not capture the global information of speech emotion, leading to a decline in emotion recognition performance. In addition, when performing emotion recognition, it is difficult to fully identify the details in the speech signal using only a simple convolutional neural network, resulting in the omission of key emotional information.

[0057] Method Implementation Examples

[0058] Reference Figure 1 The diagram shows a flowchart of a speech emotion recognition method provided by an embodiment of the present invention.

[0059] This invention provides a voice emotion recognition method, the method comprising:

[0060] Specifically, the method includes steps S1 to S7.

[0061] S1, acquire the raw speech signal.

[0062] S2, calculate the frame-level score of the original speech signal, and divide the original speech signal into high-reliability frame signals and low-reliability frame signals according to the frame-level score.

[0063] It should be noted that by calculating frame-level scores and dividing speech frames into high-reliability and low-reliability frames, it is possible to effectively distinguish between frames that are more affected by noise or interference and frames with higher quality. Thus, during feature extraction, high-reliability frames are used to extract MFCC spectral features, while low-reliability frames are used to extract prosodic features. This achieves a balance between noise suppression and full utilization of information, improves the robustness and accuracy of the emotion recognition system in speech under complex environments, and preserves emotional cues in low-reliability frames, making the overall features more comprehensive and the emotion judgment more accurate.

[0064] In one possible implementation, S2 specifically includes:

[0065] S201 calculates the spectral entropy, spectral flatness, and harmonicity of the original speech signal.

[0066] Among them, spectral entropy is used to measure the complexity or uncertainty of the spectrum of a speech signal. When the spectral energy is concentrated, the spectral entropy is low, indicating strong signal regularity. When the spectral energy is evenly distributed, the spectral entropy is high, indicating a complex signal or a lot of noise. Spectral flatness measures the flatness of the spectral energy distribution and is usually used to determine whether the signal is similar to noise (high flatness) or a tonal signal (low flatness). Harmonicity measures the strength of the periodic components in a speech signal, that is, the clarity and periodicity of speech. High harmonicity indicates strong signal periodicity and obvious pitch, while low harmonicity indicates an irregular signal or interference from noise.

[0067] S202, weighted summation of spectral entropy, spectral flatness and harmonicity, yields the frame-level score.

[0068] Specifically, by weighted summing of spectral entropy, spectral flatness, and harmonicity, a frame-level score for each frame is obtained. This allows for a comprehensive consideration of the complexity, stationarity, and periodicity of the speech signal, thereby more accurately measuring the reliability of each frame.

[0069] S203, classify the original speech signal with a frame-level score greater than a preset score as a high-reliability frame signal, and classify the original speech signal with a frame-level score less than a preset score as a low-reliability frame signal.

[0070] It should be noted that those skilled in the art can set the preset score according to actual needs, and this invention does not limit this.

[0071] S3 extracts features from the high-reliability frame signal to determine the Mel frequency cepstral features.

[0072] Among them, Mel frequency cepstral features are speech features that simulate the perceptual characteristics of the human ear. They are obtained by mapping the spectrum to the Mel scale and then performing discrete cosine transform. They can effectively represent the short-time spectral information of speech and are widely used in speech recognition and emotion recognition.

[0073] It should be noted that by extracting Mel-frequency cepstral (MFCC) features from high-reliability frame signals, the short-time spectral characteristics of speech can be accurately captured, reducing the impact of noise interference on frequency features, improving the accuracy and stability of emotion classification, and providing a reliable foundation for subsequent feature fusion and model training.

[0074] In one possible implementation, S3 specifically includes:

[0075] S301 performs a fast Fourier transform on the high-reliability frame signal to obtain the spectrum.

[0076] S302 performs Mel filtering on the spectrum to obtain the logarithmic energy.

[0077] S303, the discrete cosine transform is performed on the logarithmic energy to obtain the static cepstral characteristics.

[0078] S304 performs dynamic feature calculations on static cepstral features to obtain first-order difference features and second-order difference features.

[0079] S305, by splicing static cepstral features, first-order difference features, and second-order difference features, the Mel frequency cepstral features are obtained.

[0080] Specifically, concatenating static cepstral features, first-order difference features, and second-order difference features can simultaneously reflect the short-term spectral information, dynamic change trends, and acceleration information of speech. This multi-level feature fusion method not only preserves the basic spectral characteristics of speech but also enhances the sensitivity to subtle changes in speech emotion, thereby improving the ability of the emotion recognition model to distinguish different emotional states and the overall recognition accuracy.

[0081] The first-order difference feature and the second-order difference feature are as follows:

[0082]

[0083] ΔΔCep(i)=ΔCep(i+1)-ΔCep(i-1)

[0084] Wherein, ΔCep(i) represents the first-order differential feature of the i-th high-reliability frame signal, α represents the scaling factor, Cep(i+j) represents the static cepstral feature of the (i+j)-th high-reliability frame signal, Cep(ij) represents the static cepstral feature of the ij-th high-reliability frame signal, ΔCep(i) represents the second-order differential feature of the i-th high-reliability frame signal, ΔCep(i+1) represents the first-order differential feature of the (i+1)-th high-reliability frame signal, ΔCep(i-1) represents the first-order differential feature of the (i-1)-th high-reliability frame signal, j represents the relative displacement index, and J represents the half-window length of the differential.

[0085] S4 extracts features from low-reliability frame signals to determine prosodic features.

[0086] Among them, prosodic features describe the global characteristics of speech such as rhythm, stress, intonation, and energy changes, including zero-crossing rate, Teager energy operator, harmonic-to-noise ratio, etc., which mainly reflect the emotion and speaking style of speech, rather than relying on precise spectral details.

[0087] In one possible implementation, prosodic features specifically include: zero-crossing rate, Teager energy operator, and harmonic-to-noise ratio.

[0088]

[0089] Ψ[x f [k]]=x f [k] 2 -x f [k-1]x f [k+1]

[0090] Where ZCR represents the zero-crossing rate, sign represents the indicator function, N represents the total number of sampling points in the low-reliability frame signal, s(n) represents the discrete speech amplitude of the nth sampling point in the low-reliability frame signal, s(n-1) represents the discrete speech amplitude of the (n-1)th sampling point in the low-reliability frame signal, and x f [k] represents the amplitude of the low-reliability frame signal at sampling time k, Ψ[] represents the output of the Teager operator, x f [k-1] represents the amplitude of the low-reliability frame signal at sampling time k-1, x f [k+1] represents the amplitude of the low-reliability frame signal at sampling time k+1.

[0091] Zero-crossing rate (ZCR) represents the number of times a speech signal crosses zero amplitude per unit time, reflecting the frequency variation characteristics of speech. A high ZCR usually corresponds to noisy or silent segments, while a low ZCR corresponds to speech segments with clear pitch. The Teager energy operator is a nonlinear energy measurement method used to estimate the instantaneous energy and nonlinear characteristics of speech signals. Harmonic-to-noise ratio (HNR) measures the ratio of periodic components (harmonics) to non-periodic components (noise) in a speech signal, reflecting the clarity and sound quality of the speech. A high HNR indicates clear speech with strong periodicity, while a low HNR indicates a high proportion of noise or a hoarse voice.

[0092] In this embodiment of the invention, high-reliability frame signals are typically the portions of speech with low noise interference and clear spectra. These frames are suitable for extracting Mel-frequency cepstral features (MFCC) because MFCC accurately reflects the short-time spectral structure and formant information of speech, serving as a core representation of speech content and articulation features. If these frames are extracted together with low-reliability frames containing more noise, MFCC is easily affected by noise, leading to inaccurate spectral features and reduced emotion recognition performance. Low-reliability frame signals typically contain noise or portions with low speech energy, but these frames still retain emotional features in their prosodic information, such as pitch fluctuations, rhythm, energy changes, and harmonic-to-noise ratio. These features can supplement the emotional information that MFCC cannot fully reflect. Through this targeted extraction, the advantages of high- and low-reliability frames are fully utilized, achieving complementarity between spectral and prosodic information, thereby significantly improving the accuracy and robustness of speech emotion recognition.

[0093] S5, feature fusion is performed on Mel frequency cepstral features and prosodic features to obtain target features.

[0094] It should be noted that by fusing Mel frequency cepstral features (MFCC) and prosodic features, short-time spectral information of speech can be integrated with global rhythm, energy and intelligibility information. This can supplement the shortcomings of single features, enabling the model to capture both the micro-frequency changes of speech and the macro-expression of emotions, thereby significantly improving the accuracy and robustness of emotion recognition and enhancing the model's adaptability to different speaking styles or noisy environments.

[0095] In one possible implementation, S5 specifically includes:

[0096] S501 standardizes the cepstral and prosodic features of Mel frequency.

[0097] S502, weighted pooling is performed on the standardized Mel frequency cepstral features and prosodic features respectively to obtain pooled Mel frequency cepstral features and pooled prosodic features.

[0098] Specifically, weighted pooling is performed on the standardized Mel frequency cepstral features and prosodic features respectively. Different weights can be assigned according to the importance of the features. At the same time, local features are aggregated in the time dimension, thereby preserving key emotional information and reducing the impact of redundant noise.

[0099] The specific method for calculating the pooled Mel-frequency cepstral characteristics is as follows:

[0100]

[0101] in, F represents the pooled Mel-frequency cepstral characteristic. MFCC [i] represents the Mel frequency cepstral feature of the i-th high-reliability frame signal, η i H represents the linear weighting coefficient of the i-th high-reliability frame signal, β represents the scaling factor of the nonlinear transform, tanh represents the hyperbolic tangent function, and H... g Let γ represent the weight matrix. i δ represents the coefficients for nonlinear weighting of the i-th high-reliability frame signal, T represents the total number of high-reliability frame signals, and δ uv′ The weighting coefficients F, representing the interaction between the u-th and v′-th MFCC features. MFCC [:,u] represents the u-th dimension vector of the Mel frequency cepstral feature, ⊙ denotes element-wise multiplication, F MFCC [:,v′] represents the v′-th dimension vector of the Mel frequency cepstral feature, D m This represents the total dimension of the MFCC features.

[0102] In this embodiment of the invention, the calculation method for pooling prosodic features is the same as the calculation method for pooling Mel-frequency cepstral features.

[0103] S503, feature fusion is performed on pooled Mel frequency cepstral features and pooled prosodic features to obtain target features.

[0104] Specifically, by fusing pooled Mel frequency cepstral features with prosodic features, the final target features simultaneously contain acoustic detail information and prosodic rhythm information, thereby enhancing the speech emotion recognition model's ability to identify different emotion features and improving the accuracy and robustness of recognition.

[0105] S6. Construct a speech emotion recognition model, which specifically includes: a local feature learning block, a temporal learning module, a fully connected module, and a classification module.

[0106] In one possible implementation, the local feature learning block includes multiple LFLB units, each of which includes a convolutional layer, a batch normalization layer, an ELU activation function, and a max pooling layer.

[0107] S7. Input the target features into the speech emotion recognition model to determine the emotion category.

[0108] It should be noted that by fully inputting the fused target features into the speech emotion recognition model, the model can make full use of the acoustic features of high-reliability frames and the prosodic features of low-reliability frames to perform temporal modeling and global feature aggregation, thereby achieving accurate classification of the speaker's emotions, improving recognition accuracy and system robustness, while effectively reducing the impact of noise interference on emotion determination.

[0109] In one possible implementation, S7 specifically includes:

[0110] S701, in the local feature learning block, local feature extraction is performed on the target features to obtain the sampled feature map.

[0111] In one possible implementation, S701 specifically includes:

[0112] S7011 performs convolution operations on the target features in the convolutional layer to generate a convolutional feature map.

[0113]

[0114] Where z(p,q) represents the convolutional feature map at position (p,q), a and b represent integers defining the size of the convolutional kernel, x′(s′,t′) represents the pixel of the target feature at position (s′,t′), and w(ps′,qt′) represents the convolutional kernel weight at the offset position (ps′,qt′).

[0115] In this embodiment of the invention, a controls the size of the convolution kernel in the vertical direction (height), and b controls the size of the convolution kernel in the horizontal direction (width).

[0116] S7012, in the batch normalization layer, normalizes the convolutional feature map to obtain normalized features.

[0117]

[0118] in, Let σ represent the normalized feature of the p′-th output channel of the l-th layer network, σ represent the Sigmoid activation function, and BN represent batch normalization. This represents the bias term for the p′-th output channel of the l-th layer network. This represents the normalized feature of the q′-th output channel of the (l-1)-th layer network. This represents the convolution kernel weights in the l-th layer that map the input channel q′ to the output channel p′, and * represents the convolution operation.

[0119] S7013, in the ELU activation function, nonlinear enhancement is applied to the normalized features to obtain nonlinear features.

[0120] S7014, in the max pooling layer, performs dimensionality reduction on nonlinear features to obtain downsampled feature maps.

[0121] In this embodiment of the invention, local patterns of target features are extracted through convolutional layers, and batch normalization is used to enhance feature stability. The ELU activation function is used to introduce nonlinear expressive power, and finally max pooling is used to reduce feature dimensionality while retaining key information, thereby generating a representative sampled feature map. This process can not only effectively capture the local time-frequency structure in speech signals, but also suppress noise interference and improve the recognition accuracy and robustness of the model in subsequent time-series modeling and emotion classification.

[0122] S702, in the time series learning module, performs global modeling on the sampled feature map to obtain aggregated time series features.

[0123] It should be noted that by performing global modeling of the sampled feature map in the temporal learning module, the long-term dependencies and dynamic changes of the speech signal in the time dimension can be fully captured. This effectively integrates local features into aggregated temporal features, enhances the model's understanding of the continuity and overall trend of speech emotions, and improves the accuracy and robustness of emotion recognition.

[0124] In one possible implementation, the temporal learning module specifically includes a dimensional projection unit, an improved LSTM unit, and a global aggregation unit.

[0125] S702 specifically includes:

[0126] S7021, in the dimensional projection unit, the sampled feature map is mapped to the target dimension to obtain the mapped feature map:

[0127]

[0128] Z = AXW V

[0129]

[0130] Where A represents the attention weight matrix, softmax represents the softmax activation function, X represents the sampled feature map, and W... Q W represents the projection matrix of the query. K The projection matrix representing the bond, d k B represents the attention projection dimension. rel Z represents the relative positional bias, and W represents the global contextual feature. V The projection matrix represents the values, G represents the gate matrix, σ represents the sigmoid activation function, and W represents the projection matrix.g W′ represents the weights that map from the sampled feature map to the gate. g b represents the weights that map from global context features to the gate. g Indicates the gated bias, L represents the linear projection result, and W... L W represents the linear projection matrix of the sampled feature map. Z The linear projection matrix representing the global context features, b L U represents the bias term of the linear projection, Q represents the second-order interaction feature, and U represents the bias term of the linear projection. r V represents the low-rank projection matrix of the r-th rank decomposition of the sampled feature map, where R represents the rank and V r S represents the low-rank projection matrix of the r-th rank decomposition of the context features. r Let r be the second-order inter-matrix of the rank decomposition. Represents convolutional branch features, Conv1D causal (X;K,b c ) represents causal one-dimensional convolution, K represents the convolution kernel weights, and b c W represents the convolution bias. C This represents the projection matrix of the convolution branch. denoted as the mapping feature map, LayerNorm represents layer normalization, ⊙ represents element-wise multiplication, and T represents transpose.

[0131] Specifically, by mapping the sampled feature map to the target dimension through the dimensional projection unit, the feature representation of each channel and time step can be unified, making local features from different sources or of different types comparable and fused in the same space, thus reducing feature redundancy.

[0132] In S7022, the improved LSTM unit performs global dependency modeling on the mapped feature map to obtain temporal features:

[0133] f t =σ(W f [x t ,h t-1 C t-1 ]+b f )

[0134]

[0135] O t =σ(W o [x t ,h t-1 ]+b o )

[0136] P t =tanh(W p [x t ,h t-1 ]+bp )

[0137] h t =O t ·tanh(C t )+(1-O t )·P t

[0138] Among them, f t W represents the forgetting coefficient at time t. f Let x represent the forget gate weight matrix. t H represents the mapping feature map at time t. t-1 Let C represent the hidden state at time t-1. t-1 b represents the cell state at time t-1. f Indicates the forget gate bias term. Let W represent the updated cell state at time t, tanh represent the hyperbolic tangent function, and W represent the cell state at time t. a The weight matrix representing the candidate state, b a C represents the candidate state bias term. t O represents the cell state at time t. t W represents the output gate at time t. o b represents the output gate weight matrix. o P represents the output gate bias term. t W represents the fast response component at time t. p b represents the weight matrix of the input through-branch. p h represents the bias vector of the input pass-through branch. t This represents the hidden state at time t, i.e., the time sequence feature.

[0139] Specifically, by using an improved LSTM unit to perform global dependency modeling on the mapped feature map, the long-range correlation and dynamic change features of the speech signal in the time dimension can be captured. This effectively preserves the temporal patterns and intonation changes in emotional expression, enhances the model's ability to perceive subtle emotional changes, and improves the continuity and stability of temporal features.

[0140] S7023, in the global aggregation unit, performs global time series feature aggregation on the time series features to obtain aggregated time series features.

[0141] Specifically, by integrating information from the entire time series to generate aggregated temporal features, speech emotion information spanning a long period can be uniformly encoded, preserving global emotion patterns and contextual relationships, reducing the impact of local fluctuations on emotion judgment, thereby improving the accuracy and stability of emotion recognition.

[0142] In S703, the aggregated temporal features are mapped to the emotion category space in the fully connected module to obtain the category score vector.

[0143] In this embodiment of the invention, the aggregated temporal features are flattened into a one-dimensional vector, and then linearly mapped to the weight matrix through one or more fully connected layers. Each output node corresponds to an emotion category. Through matrix multiplication and bias addition operations, the multidimensional information of the input features is mapped to the emotion category space to generate a category score vector.

[0144] S704, in the classification module, the category score vector is converted into the probability distribution of each emotion category to determine the emotion category.

[0145] In this embodiment of the invention, the softmax function is used to convert the category score vector into a probability distribution for each emotion category.

[0146] The beneficial effects of this invention are as follows:

[0147] In this embodiment of the invention, the original speech signal is divided into frame-level reliability segments. Different feature extraction strategies can be adopted for speech information with different reliability levels, ensuring the accuracy of the spectral features of high-reliability frames while making full use of the prosodic information of low-reliability frames to avoid the loss of emotional information. By extracting features from high-reliability and low-reliability frame signals respectively, the different information characteristics of speech signals can be fully utilized, improving the accuracy and robustness of feature expression. The extracted features are spliced ​​and fused to unify the scale of different features, reduce redundant information interference, and avoid information imbalance. Through the local feature learning block, temporal learning module, fully connected module, and classification module of the speech emotion recognition model, short-term transient emotions and long-term gradual emotions in speech are fully captured, the probability distribution of emotion categories is determined, and accurate emotion determination is achieved, significantly improving the accuracy of speech emotion recognition and its stability in complex environments.

[0148] System Implementation Examples

[0149] Reference manual attached Figure 2 The diagram shows a structural schematic of a voice emotion recognition system provided in an embodiment of the present invention.

[0150] The present invention proposes a voice emotion recognition system 30, comprising: a memory 303 and a processor 301.

[0151] The memory 303 stores an application program adapted to be executed by the processor 301 to implement the big data-based oil equipment fault prediction method of the method embodiment.

[0152] The big data-based oil equipment fault prediction system 30 includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302.

[0153] The structure of the voice emotion recognition system 30 does not constitute a limitation on the embodiments of the present invention.

[0154] Processor 301 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0155] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI bus or an EISA bus, etc. Bus 302 may be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not mean that there is only one bus or one type of bus.

[0156] The memory 303 may be a ROM or other type of static storage device capable of storing static information and instructions, RAM or other type of dynamic storage device capable of storing information and instructions, or it may be an EEPROM, CD-ROM or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0157] Computer-readable storage medium embodiments

[0158] The present invention proposes a computer-readable storage medium having a computer program stored thereon, the computer program being loadable and executed by a processor for a speech emotion recognition method of the first aspect.

[0159] The applicant of this invention has provided a detailed description of the embodiments of the invention in conjunction with the accompanying drawings. However, those skilled in the art should understand that the above embodiments are merely preferred embodiments of the invention. The detailed description is only intended to help readers better understand the spirit of the invention and is not intended to limit the scope of protection of the invention. On the contrary, any improvements or modifications made based on the inventive spirit of the invention should fall within the scope of protection of the invention.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A voice emotion recognition method, characterized in that, The voice emotion recognition method includes: S1, acquire the original speech signal; S2, calculate the frame-level score of the original speech signal, and divide the original speech signal into high-reliability frame signals and low-reliability frame signals according to the frame-level score; S3, perform feature extraction on the high-reliability frame signal to determine the Mel frequency cepstral features; S4, perform feature extraction on the low-reliability frame signal to determine prosodic features; S5, perform feature fusion on the Mel frequency cepstral features and the prosodic features to obtain the target features; S6, Construct a speech emotion recognition model, wherein the speech emotion recognition model specifically includes: a local feature learning block, a temporal learning module, a fully connected module, and a classification module; S7. Input the target features into the voice emotion recognition model to determine the emotion category.

2. The speech emotion recognition method according to claim 1, characterized in that, S2 specifically includes: S201, Calculate the spectral entropy, spectral flatness, and harmonicity of the original speech signal; S202, the weighted sum of the spectral entropy, the spectral flatness, and the harmonicity is obtained to get the frame-level score; S203, the original speech signal with a frame-level score greater than a preset score is classified as the high-reliability frame signal, and the original speech signal with a frame-level score less than the preset score is classified as the low-reliability frame signal.

3. The speech emotion recognition method according to claim 1, characterized in that, S3 specifically includes: S301, Perform a fast Fourier transform on the high-reliability frame signal to obtain the spectrum; S302, Perform Mel filtering on the spectrum to obtain the logarithmic energy; S303, Perform discrete cosine transform on the logarithmic energy to obtain static cepstral features; S304, Perform dynamic feature calculation on the static cepstral features to obtain first-order difference features and second-order difference features; S305, the static cepstral feature, the first-order difference feature, and the second-order difference feature are spliced ​​together to obtain the Mel frequency cepstral feature.

4. The speech emotion recognition method according to claim 1, characterized in that, The prosodic features specifically include: zero-crossing rate, Teager energy operator, and harmonic-to-noise ratio.

5. The speech emotion recognition method according to claim 1, characterized in that, S5 specifically includes: S501, Standardize the Mel frequency cepstral features and the prosodic features; S502, weighted pooling is performed on the standardized Mel frequency cepstral features and prosodic features respectively to obtain pooled Mel frequency cepstral features and pooled prosodic features; S503, the pooled Mel frequency cepstral features and the pooled prosodic features are fused to obtain the target features.

6. The speech emotion recognition method according to claim 1, characterized in that, The local feature learning block includes multiple LFLB units, each of which includes a convolutional layer, a batch normalization layer, an ELU activation function, and a max pooling layer.

7. The speech emotion recognition method according to claim 1, characterized in that, Specifically, S7 includes: S701, In the local feature learning block, local feature extraction is performed on the target feature to obtain a sampled feature map; S702, in the time-series learning module, global modeling is performed on the sampled feature map to obtain aggregated time-series features; S703, in the fully connected module, the aggregated temporal features are mapped to the emotion category space to obtain a category score vector; S704, in the classification module, the category score vector is converted into an emotion category probability distribution to determine the emotion category.

8. The speech emotion recognition method according to claim 7, characterized in that, Specifically, S701 includes: S7011, In the convolutional layer of the local feature learning block, the target feature is convolved to generate a convolutional feature map; S7012, in the batch normalization layer of the local feature learning block, the convolutional feature map is normalized to obtain normalized features; S7013, In the ELU activation function of the local feature learning block, the normalized features are nonlinearly enhanced to obtain nonlinear features; S7014, in the max pooling layer of the local feature learning block, the nonlinear features are reduced in dimensionality to obtain the sampled feature map.

9. The speech emotion recognition method according to claim 7, characterized in that, The temporal learning module specifically includes a dimensional projection unit, an improved LSTM unit, and a global aggregation unit; Specifically, S702 includes: S7021, In the dimension projection unit, the sampled feature map is mapped to the target dimension to obtain a mapped feature map; S7022, In the improved LSTM unit, global dependency modeling is performed on the mapped feature map to obtain temporal features; S7023, In the global aggregation unit, the time series features are aggregated globally to obtain the aggregated time series features.

10. A voice emotion recognition system, characterized in that, include: Memory and processor; The memory stores an application program adapted to be executed by the processor to implement the voice emotion recognition method according to any one of claims 1 to 9.

Images